In recent years, deep learning has been widely explored on graph structured data, such as chemical compounds, protein structures, financial networks, and social networks[1, 2, 3]. Remarkable success has been achieved by generalizing deep neural networks from grid-like data to graphs [4, 5, 6, 7], resulting in the development of various graph neural networks (GNNs), like graph convolutional network (GCN) , GraphSAGE , graph attention network (GAT) , jumping knowledge network (JK) , and graph isomorphism networks (GINs) . They are able to learn representations for each node in graphs and have set new performance records on tasks like node classification and link prediction . In order to extend the success to graph representation learning, graph pooling is required, which takes node representations of a graph as inputs and outputs the corresponding graph representation.
While pooling is common in deep learning on grid-like data, it is challenging to develop graph pooling approaches due to the special properties of graphs. First, the number of nodes varies in different graphs, while the graph representations are usually required to have the same fixed size to fit into other machine learning models. Therefore, graph pooling should be capable of handling the variable number of node representations as inputs and producing fixed-sized graph representations. Second, unlike images and texts where we can order pixels and words according to the spatial structural information, there is no inherent ordering relationship among nodes in graphs. Indeed, we can set pseudo indices for nodes in a graph. However, an isomorphism of the graph may change the order of the indices. As isomorphic graphs should have the same graph representation, it is required for graph pooling to create the same output by taking node representations in any order as inputs.
Some previous studies employ simple methods such as averaging and summation as graph pooling [14, 15, 12]. However, averaging and summation ignore the feature correlation information, hampering the overall model performance . Other studies have proposed advanced graph pooling methods, including DiffPool , SortPool , TopKPool , SAGPool , and EigenPool . DiffPool maps nodes to a pre-defined number of clusters but is hard to train. EigenPool
involves the computation of eigenvectors, which is slow and expensive.SortPool, SAGPool and TopKPool rely on the top- sorting to select a fixed number () of nodes and order them, during which the information from unselected nodes is discarded. It is worth noting that all the existing graph pooling methods only collect first-order statistics .
In this work, we propose to use second-order pooling as graph pooling. Compared to existing graph pooling methods, second-order pooling naturally solves the challenges of graph pooling and is more powerful with its ability of using information from all nodes and collecting second-order statistics. We analyze the practical problems in directly using second-order pooling with GNNs. To address the problems, we propose two novel and effective global graph pooling approaches based on second-order pooling; namely, bilinear mapping and attentional second-order pooling. In addition, we extend attentional second-order pooling to hierarchical graph pooling for more flexible use in GNNs. We perform thorough experiments on ten graph classification benchmark datasets. The experimental results show that our methods improve the performance significantly and consistently.
2 Related Work
In this section, we review two categories of existing graph pooling methods in Section 2.1. Then in Section 2.2, we introduce what second-order statistics are, as well as their applications in both transitional machine learning and deep learning. In addition, we discuss the motivation of using second-order statistics in graph representation learning.
2.1 Graph Pooling: Global versus Hierarchical
Existing graph pooling methods can be divided into two categories according to their roles in graph neural networks (GNNs) for graph representation learning. One is global graph pooling, also known as graph readout operation [12, 19]. The other is hierarchical graph pooling, which is used to build hierarchical GNNs. We explain the details of the two categories and provide examples. In addition, we discuss advantages and disadvantages of the two categories.
Global graph pooling is typically used to connect embedded graphs outputted by GNN layers with classifiers for graph classification. Given a graph, GNN layers produce node representations, where each node is embedded as a vector. Global graph pooling is applied after GNN layers to process node representations into a single vector as the graph representation. A classifier takes the graph representation and performs graph classification. The “global” here refers to the fact that the output of global graph pooling encodes the entire graph. Global graph pooling is usually used only once in GNNs for graph representation learning. We call such GNNs as flat GNNs, in contrast to hierarchical GNNs. The most common global graph pooling methods include averaging and summation[14, 15, 12].
Hierarchical graph pooling is more similar to pooling in computer vision tasks. The output of hierarchical graph pooling is a pseudo graph with fewer nodes than the input graph. It is used to build hierarchical GNNs, where hierarchical graph pooling is used several times between GNN layers to gradually decrease the number of nodes. The most representative hierarchical graph pooling methods are DiffPool , SortPool , TopKPool , SAGPool , and EigenPool . A straightforward way to use hierarchical graph pooling for graph representation learning is to reduce the number of nodes to one. Then the resulted single vector is treated as the graph representation. Besides, there are two other ways to generate a single vector from the pseudo graph outputted by hierarchical graph pooling. One is introduced in SAGPool , where global and hierarchical graph pooling are combined. After each hierarchical graph pooling, global graph pooling with an independent classifier is employed. The final prediction is an average of all classifiers. On the other hand, SortPool 
directly applies convolutional neural networks (CNNs) to reduce the number of nodes to one. In particular, it takes advantage of a property of the pseudo graph outputted by hierarchical graph pooling. That is, the pseudo graph is a graph with a fixed number of nodes and there is an inherent ordering relationship among nodes determined by the trainable parameters in the hierarchical graph pooling. Therefore, common deep learning methods like convolutions can be directly used. In fact, we can simply concatenate node presentations following the inherent order as the graph representation.
Given this property, most hierarchical graph pooling methods can be flexibly used as global graph pooling, with the three ways introduced above. For example, SortPool  is used to build flat GNNs and applied only once after all GNN layers. While the idea of learning hierarchical graph representations makes sense, hierarchical GNNs do not consistently outperform flat GNNs . In addition, with advanced techniques like jumping knowledge networks (JK-Net)  to address the over-smoothing problem of GNN layers , flat GNNs can go deeper and achieve better performance than hierarchical GNNs .
In this work, we first focus on global graph pooling as second-order pooling naturally fits this category. Later, we extend one of our proposed graph pooling methods to hierarchical graph pooling in Section 3.6.
2.2 Second-Order Statistics
In statistics, the -order statistics refer to functions which use the -th power of samples. Concretely, consider samples
. The first and second moments,i.e., the mean
and variance, are examples of first and second-order statistics, respectively. If each sample is a vector, the covariance matrix is an example of second-order statistics. In terms of graph pooling, it is easy to see that existing methods are based on first-order statistics .
Second-order statistics have been widely explored in various computer vision tasks, such as face recognition, image segmentation, and object detection. In terms of traditional machine learning, the scale-invariant feature transform (SIFT) algorithm utilizes second-order statistics of pixel values to describe local features in images and has become one of the most popular image descriptors. Tuzel et‘ al.[24, 25] use covariance matrices of low-level features with boosting for detection and classification. The Fisher encoding  applies second-order statistics for recognition as well. Carreira et al.  employs second-order pooling for semantic segmentation. With the recent advances of deep learning, second-order pooling is also used in CNNs for fine-grained visual recognition  and visual question answering [29, 30, 31].
Many studies motivates the use of second-order statistics as taking advantage of the Riemannian geometry of the space of symmetric positive definite matrices [32, 25, 27]. In these studies, certain regularizations are cast to guarantee that the applied second-order statistics are symmetric positive definite [33, 34]. Other work relates second-order statistics to orderless texture descriptors for images [26, 28].
In this work, we propose to incorporate second-order statistics in graph representation learning. Our motivations lie in three aspects. First, second-order pooling naturally fits the goal and requirements of graph pooling, as discussed in Sections 3.1 and 3.2. Second, second-order pooling is able to capture the correlations among features, as well as topology information in graph representation learning, as demonstrated in Section 3.2. Third, our proposed graph pooling methods based on second-order pooling are related to covariance pooling [24, 25, 33, 34] and attentional pooling  used in computer vision tasks, as pointed out in Section 3.5. In addition, we show that both covariance pooling and attentional pooling have certain limitations when employed in graph representation learning, and our proposed methods appropriately address them.
3 Second-Order Pooling for Graphs
In this section, we introduce our proposed second-order pooling methods for graph representation learning. First, we formally define the aim and requirements of graph pooling in Section 3.1. Then we propose to use second-order pooling as graph pooling, analyze its advantages, and point out practical problems when directly using it with GNNs in Section 3.2. In order to address the problems, we propose two novel second-order pooling methods for graphs in Sections 3.3 and 3.4, respectively. Afterwards, we discuss why our proposed methods are more suitable as graph pooling compared to two similar pooling methods in image tasks in Section 3.5. Finally, while both methods focus on global graph pooling, we extend second-order pooling to hierarchical graph pooling in Section 3.6.
3.1 Properties of Graph Pooling
Consider a graph represented by its adjacency matrix and node feature matrix , where is the number of nodes in and is the dimension of node features. The node features may come from node labels or node degrees. Graph neural networks (GNNs) are known to be powerful in learning good node representation matrix from and :
where rows of , , are representations of nodes, and depends on the architecture of GNNs. The task that we focus on in this work is to obtain a graph representation vector from , which is then fed into a classifier to perform graph classification:
where is the graph pooling function and is the dimension of . Here, means that the information from can be optionally used in graph pooling. For simplicity, we omit it in the following discussion.
Note that must satisfy two requirements to serve as graph pooling. First, should be able to take with variable number of rows as the inputs and produce fixed-sized outputs. Specifically, different graphs may have different number of nodes, which means that is a variable. On the other hand, is supposed to be fixed to fit into the following classifier.
Second, should output the same when the order of rows of changes. This permutation invariance property is necessary to handle isomorphic graphs. To be concrete, if two graph and are isomorphic, GNNs will output the same multiset of node representations [16, 12]. That is, there exists a permutation matrix such that , for and . However, the graph representation computed by should be the same, i.e., if .
3.2 Second-Order Pooling
We start by introducing the definition of second-order pooling.
Definition. Given , second-order pooling (SOPool) is defined as
In terms of graph pooling, we can view as an -dimensional graph representation vector by simply flattening the matrix. Another way to transform the matrix into a vector is discussed in Section 3.4. Note that, as long as SOPool meets the two requirements, the way to transform the matrix into a vector does not affect its eligibility as graph pooling.
Now let us check the two requirements.
Proposition 1. SOPool always outputs an matrix for , regardless of the value of .
The result is obvious since the dimension of does not depend on . ∎
Proposition 2. SOPool is invariant to permutation so that it outputs the same matrix when the order of rows of changes.
Consider , where is a permutation matrix. Note that we have for any permutation matrix. Therefore, it is easy to derive
This completes the proof. ∎
In addition to satisfying the requirements of graph pooling, SOPool is capable of capturing second-order statistics, which are much more discriminative than first-order statistics computed by most other graph pooling methods [27, 28, 29]. In detail, the advantages can be seen from two aspects. On one hand, we can tell from that, for each node representation , the features interact with each other, enabling the correlations among features to be captured. On the other hand, topology information is encoded as well. Specifically, we view as , where , . The vector encodes the spatial distribution of the -th feature in the graph. Based on this view, is able to capture the topology information.
However, we point out that the direct application of second-order pooling in GNNs leads to practical problems. The direct way to use second-order pooling as graph pooling is represented as
That is, we apply SOPool on and flatten the output matrix into an -dimensional graph representation vector. However, it causes an explosion in the number of training parameters in the following classifier when is large, making the learning process harder to converge and easier to overfit. While each layer in a GNN usually has outputs with a small number of hidden units (e.g. 16, 32, 64), it has been pointed out that graph representation learning benefits from using information from outputs of all layers, obtaining better performance and generalization ability . It is usually achieved by concatenating outputs across all layers in a GNN [16, 12]. In this case, has a large final , making direct use of second-order pooling infeasible. For example, if a GNN has 5 layers and each layer’s outputs have 32 hidden units, becomes . Suppose is sent into a 1-layer fully-connected classifier for graph categories in a graph classification task. It results in training parameters, which is excessive. We omit the bias term for simplicity.
3.3 Bilinear Mapping Second-Order Pooling
To address the above problem, a straightforward solution is to reduce in before . Based on this, our first proposed graph pooling method, called bilinear mapping second-order pooling (SOPool), employs a linear mapping on to perform dimensionality reduction. Specifically, it is defined as
where and is a trainable matrix representing a linear mapping. Afterwards, we follow the same process to flatten the matrix and obtain an -dimensional graph representation vector:
Figure 1 provides an illustration of the above process. By selecting an appropriate , the bilinear mapping second-order pooling does not suffer from the excessive number of training parameters. Taking the example above, if we set , the total number of parameters in SOPool and a following 1-layer fully-connected classifier is , which is much smaller than .
3.4 Attentional Second-Order Pooling
Our second proposed graph pooling method tackles with the problem by exploring another way to transform the matrix computed by SOPool into the graph representation vector, instead of simply flattening. Similarly, we use a linear mapping to perform the transformation, defined as
where is a trainable vector. It is interesting to note that , which is similar to the sentence attention in . To be concrete, consider a word embedding matrix for a sentence, where is the number of words and is the dimension of word embeddings. The sentence attention is defined as
where is a trainable vector and is the resulted sentence embedding. Note that Eqn. (9) is the Softmax function and serves as a normalization function . Rewriting the sentence attention into matrix form, we have . The only difference between the computation of and that of is the normalization function. Therefore, we name our second proposed graph pooling method as attentional second-order pooling (SOPool), defined as
where is a trainable vector. It is illustrated in Figure 1. We take the same example above to show that SOPool reduces the number of training parameters. The total number of parameters in SOPool and a following 1-layer fully-connected classifier is just , significantly reducing the amount of parameters compared to .
3.5 Relationships to Covariance Pooling and Attentional Pooling
The experimental results in Section 4 show that both our proposed graph pooling methods achieve better performance significantly and consistently than previous studies. However, we note that, there are pooling methods in image tasks that have similar computation processes to our proposed methods, although they have not been developed based on second-order pooling. In this section, we point out the key differences between these methods and ours and show why they matter in graph representation learning.
Note that images are usually processed by deep neural networks into feature maps , where , , are the height, width, and number of feature maps, respectively. Following [35, 33, 34], we reshape into the matrix , where and so that different pooling methods can be compared directly.
Covariance pooling. Covariance pooling (CovPool) [24, 25, 33, 34] has been widely explored in image tasks, such as image categorization, facial expression recognition, and texture classification. Recently, it has also been explored in GNNs . The definition is
where is the -dimensional all-one vector and is the mean of rows of . It differs from SOPool defined in Eqn. (3) only in whether to subtract the mean. However, subtracting the mean makes CovPool less powerful in terms of distinguishing graphs with repeating node embeddings , which may cause the performance loss. Figure 2(a) gives an example of this problem.
where is a trainable vector. It differs from SOPool only in the Softmax function. We show that the Softmax
function leads to similar problems as other normalization functions, such as mean and max-pooling. Figure 2 provides examples in which AttnPool does not work.
To conclude, our methods derived from second-order pooling are more suitable as graph pooling. We compare these pooling methods through experiments in Section 4.3. The results show that CovPool and AttnPool suffer from significant performance loss on some datasets.
3.6 Multi-Head Attentional Second-Order Pooling
The proposed SOPool and SOPool belong to the global graph pooling category. As discussed in Section 2.1, they are used in flat GNNs after all GNN layers and output the graph representation for the classifier. While flat GNNs outperform hierarchical GNNs in most benchmark datasets , developing hierarchical graph pooling is still desired, especially for large graphs [17, 20, 19]. Therefore, we explore a hierarchical graph pooling method based on second-order pooling.
Unlike global graph pooling, hierarchical graph pooling outputs multiple vectors corresponding to node representations in the pooled graph. In addition, hierarchical graph pooling has to update the adjacency matrix to indicate how nodes are connected in the pooled graph. To be specific, given the adjacency matrix and node representation matrix , a hierarchical graph pooling function can be written as
where and . Here,
is a hyperparameter determining the number of nodes in the pooled graph. Note that Eqn. (14) does not conflict with Eqn. (2), as we can always transform into a vector , as discussed in Section 2.1.
We note that the proposed SOPool in Section 3.4 is closely related to the attention mechanism and can be easily extended to a hierarchical graph pooling method based on the multi-head technique in the attention mechanism [37, 10]. The multi-head technique means that multiple independent attentions are performed on the same inputs. Then the outputs of multiple attentions are then concatenated together. Based on this insight, we propose multi-head attentional second-order pooling (SOPool), defined as
where is a trainable matrix. To illustrate its relationship to the multi-head technique, we can equivalently write it as
The multi-head technique enables SOPool to output the node representation matrix for the pooled graph. We now describe how to update the adjacency matrix. In particular, we employ a contribution matrix in updating the adjacency matrix. The contribution matrix is a matrix, whose entries indicate how nodes in the input graph contribute to nodes in the pooled graph. In SOPool, we can simply let . With the contribution matrix , the corresponding adjacency matrix of the pooled graph can be computed as
The proposed SOPool is closely related to DiffPool . The contribution matrix corresponds to the assignment matrix in DiffPool. However, DiffPool applied GNN layers with normalization on to obtain , preventing the explicit use of second-order statistics. In the experiments, we evaluate SOPool as both global and hierarchical graph pooling methods, in flat and hierarchical GNNs, respectively.
We conduct thorough experiments on graph classification tasks to show the effectiveness of our proposed graph pooling methods, namely bilinear mapping second-order pooling (SOPool), attentional second-order pooling (SOPool), and multi-head attentional second-order pooling (SOPool). Section 4.1 introduces the datasets, baselines, and experimental setups for reproduction. The following sections aim at evaluating our proposed methods in different aspects, by answering the questions below:
Can GNNs with our proposed methods achieve improved performance in graph classification tasks? Section 4.2 provides the comparison results between our methods and existing methods in graph classification tasks.
Do our proposed methods outperform existing global graph pooling methods with the same flat GNN architecture? The ablation studies in Section 4.3 compare different graph pooling methods with the same GNN, eliminating the influences of different GNNs. In particular, we use hierarchical graph pooling methods as global graph pooling methods in this experiment, including SOPool.
Is the improvement brought by our proposed method consistent with various GNN architectures? Section 4.4 shows the performance of the proposed SOPool and SOPool with different GNNs.
Is SOPool effective as hierarchical graph pooling methods? We compare SOPool with other hierarchical graph pooling methods in the same hierarchical GNN architecture in Section 4.5.
4.1 Experimental Setup
Reproducibility. The code used in our experiments is available at https://github.com/divelab/sopool. Details of datasets and parameter settings are described below.
Datasets. We use ten graph classification datasets from , including five bioinformatics datasets (MUTAG, PTC, PROTEINS, NCI1, DD) and five social network datasets (COLLAB, IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY, REDDIT-MULTI5K). Note that only bioinformatics datasets come with node labels. Below are the detailed descriptions of datasets:
MUTAG is a bioinformatics dataset of 188 graphs representing nitro compounds. Each node is associated with one of 7 discrete node labels. The task is to classify each graph by determining whether the compound is mutagenic aromatic or heteroaromatic .
PTC  is a bioinformatics dataset of 344 graphs representing chemical compounds. Each node comes with one of 19 discrete node labels. The task is to predict the rodent carcinogenicity for each graph.
PROTEINS  is a bioinformatics dataset of 1,113 graph structures of proteins. Nodes in the graphs refer to secondary structure elements (SSEs) and have discrete node labels indicating whether they represent a helix, sheet or turn. And edges mean that two nodes are neighbors along the amino-acid sequence or in space. The task is to predict the protein function for each graph.
NCI1  is a bioinformatics dataset of 4,110 graphs representing chemical compounds. It contains data published by the National Cancer Institute (NCI). Each node is assigned with one of 37 discrete node labels. The graph classification label is decided by NCI anti-cancer screens for ability to suppress or inhibit the growth of a panel of human tumor cell lines.
COLLAB is a scientific collaboration dataset of 5,000 graphs corresponding to ego-networks generated using the method in . The dataset is derived from 3 public collaboration datasets . Each ego-network contains different researchers from each field and is labeled by the corresponding field. The three fields are High Energy Physics, Condensed Matter Physics, and Astro Physics.
IMDB-BINARY is a movie collaboration dataset of 1,000 graphs representing ego-networks for actors/actresses. The dataset is derived from collaboration graphs on Action and Romance genres. In each graph, nodes represent actors/actresses and edges simply mean they collaborate the same movie. The graphs are labeled by the corresponding genre and the task is to identify the genre for each graph.
IMDB-MULTI is multi-class version of IMDB-BINARY. It contains 1,500 ego-networks and has three extra genres, namely, Comedy, Romance and Sci-Fi.
REDDIT-BINARY is a dataset of 2,000 graphs where each graph represents an online discussion thread. Nodes in a graph correspond to users appearing in the corresponding discussion thread and an edge means that one user responded to another. Datasets are crawled from top submissions under four popular subreddits, namely, IAmA, AskReddit, TrollXChromosomes, atheism. Among them, AmA and AskReddit are question/answer-based subreddits while TrollXChromosomes and atheism are discussion-based subreddits, forming two classes to be classified.
REDDIT-MULTI5K is a similar dataset as REDDIT-BINARY, which contains 5,000 graphs. The difference lies in that REDDIT-MULTI5K crawled data from five different subreddits, namely, worldnews, videos, AdviceAnimals, aww and mildlyinteresting. And the task is to identify the subreddit of each graph instead of determining the type of subreddits.
DD  is a bioinformatics dataset of 1,178 graph structures of proteins. Nodes in the graphs represent amino acids. And edges connect nodes that are less than 6 ngstroms apart. The task is a two-way classification task between enzymes and non-enzymes. DD is only used in Section 4.5. The average number of nodes in DD is 284.3.
|# nodes (max)||28||109||620||111||492||136||89||3783||3783|
|# nodes (avg.)||18.0||25.6||39.1||29.9||74.5||19.8||13.0||429.6||508.5|
|RW ||79.22.1||57.91.3||74.20.4||1 day||-||-||-||-||-|
|GIN-0 + Sum/Avg ||89.45.6||64.67.0||76.22.8||82.71.7||80.21.9||75.15.1||52.32.8||92.42.5||57.51.5|
|GIN-0 + SOPool||93.64.1||72.96.2||79.43.2||82.81.4||81.11.8||78.14.0||54.32.6||91.72.7||58.31.4|
|GIN-0 + SOPool||95.34.4||75.04.3||80.12.7||83.61.4||79.91.9||78.44.7||54.63.6||89.63.3||58.41.6|
|GIN-0 + SOPool||95.25.4||74.45.5||79.53.1||84.51.3||77.61.9||78.52.8||54.32.1||90.00.8||55.82.2|
|GIN-0 + Sum/Avg||89.45.6||64.67.0||76.22.8||82.71.7||80.21.9||75.15.1||52.32.8||92.42.5||57.51.5|
|GIN-0 + DiffPool||94.84.8||66.17.7||78.83.1||76.61.3||75.32.2||74.44.0||50.13.2||-||-|
|GIN-0 + SortPool||95.23.9||69.56.3||79.23.0||78.92.7||78.21.6||77.52.7||53.12.9||81.64.6||48.44.8|
|GIN-0 + TopKPool||94.73.5||68.46.4||79.12.2||79.61.7||79.62.1||77.85.1||53.72.8||-||-|
|GIN-0 + SAGPool||93.93.3||69.06.6||78.43.1||79.02.8||78.91.7||77.82.9||53.12.8||-||-|
|GIN-0 + AttnPool||93.25.8||71.28.0||77.53.3||80.62.1||81.82.2||77.14.4||53.82.5||92.52.3||57.91.7|
|GIN-0 + SOPool||93.64.1||72.96.2||79.43.2||82.81.4||81.11.8||78.14.0||54.32.6||91.72.7||58.31.4|
|GIN-0 + CovPool||95.33.7||73.35.1||80.12.2||83.51.9||79.31.8||72.15.1||47.82.7||90.33.6||58.41.7|
|GIN-0 + SOPool||95.34.4||75.04.3||80.12.7||83.61.4||79.91.9||78.44.7||54.63.6||89.63.3||58.41.6|
|GIN-0 + SOPool||95.25.4||74.45.5||79.53.1||84.51.3||77.61.9||78.52.8||54.32.1||90.00.8||55.82.2|
More statistics of these datasets are provided in the “datasets” section of Table I. The input node features are different for different datasets. For bioinformatics datasets, the nodes have categorical labels as input features. For social network datasets, we create node features. To be specific, we set all node feature vectors to be the same for REDDIT-BINARY and REDDIT-MULTI5K 
. And for the other social network datasets, we use one-hot encoding of node degrees as features.
Configurations. In Sections 4.2, 4.3 and 4.4, the flat GNNs we use with our proposed graph pooling methods are graph isomorphism networks (GINs) . The original GINs employ averaging or summation (Sum/Avg) as the graph pooling function; specifically, summation on bioinformatics datasets and averaging on social datasets. We replace averaging or summation with our proposed graph pooling methods and keep other parts the same. There are seven variants of GINs, two of which are equivalent to graph convolutional network (GCN)  and GraphSAGE , respectively. In Sections 4.2 and 4.3, we use GIN-0 with our methods. In Section 4.4, we examine our methods with all variants of GINs. Details of all variants can be found in Section 4.4.
The hierarchical GNNs used in Section 4.5 follow the hierarchical architecture in , allowing direct comparisons. To be specific, each block is composed of one GNN layer followed by a hierarchical graph pooling. After each hierarchical pooling, a classifier is used. The final prediction is the combination of all classifiers.
, model performance is evaluated using 10-fold cross-validation and reported as the average and standard deviation of validation accuracies across the 10 folds. For the flat GNNs, we follow the same training process in47]. For the hierarchical GNNs, we follow the the same training process in . There are three blocks in total. Dropout  is applied in the classifiers. The Adam optimizer 
is used with the learning rate initialized as 0.01 and decayed by 0.5 every 50 epochs. The number of total epochs is selected according to the best cross-validation accuracy. We tune the number of hidden units (16, 32, 64) and the batch size (32, 128) using grid search.
Baselines. We compare our methods with various graph classification models as baselines, including both kernel-based and GNN-based methods. The kernel-based methods are graphlet kernel (GK) , random walk kernel (RW) , Weisfeiler-Lehman subtree kernel (WL) , deep graphlet kernel (DGK) , and anonymous walk embeddings (AWE) . Among them, DGK and AWE use deep learning methods as well. The GNN-based methods are diffusion-convolutional neural network (DCNN) , PatchScan , ECC , deep graph CNN (DGCNN) , differentiable pooling (DiffPool) , graph capsule CNN (GCAPS-CNN) , self-attention graph pooling (SAGPool) , GIN , and eigenvector-based pooling (EigenGCN) . We report the performance of these baselines provided in [16, 38, 12, 17, 19, 20].
4.2 Comparison with Baselines
The comparison results between our methods and baselines are reported in Table I. GIN-0 equipped with our proposed graph pooling methods, “GIN-0 + SOPool”, “GIN-0 + SOPool”, and “GIN-0 + SOPool”, outperform all the baselines significantly on seven out of nine datasets. On NCI1, WL has better performance than all GNN-based models. However, “GIN-0 + SOPool” is the second best model and has improved performance over other GNN-based models. On REDDIT-BINARY, our methods achieve comparable performance to the best one.
It is worth noting that the baseline “GIN-0 + Sum/Avg” is the previous state-of-the-art model . Our methods differ from it only in the graph pooling functions. The significant improvement demonstrates the effectiveness of our proposed graph pooling methods. In the next section, we compare our methods with other graph pooling methods by fixing the GNN before graph pooling to GIN-0, in order to eliminate the influences of different GNNs.
4.3 Ablation Studies in Flat Graph Neural Networks
We perform ablation studies to show that our proposed methods are superior to other global graph pooling methods under a fair setting. Starting from the baseline “GIN-0 + Sum/Avg”, we replace Sum/Avg with different graph pooling methods and keep all other configurations unchanged. The graph pooling methods we include are DiffPool , SortPool from DGCNN , TopKPool from Graph U-Net , SAGPool , and CovPool and AttnPool described in Section 3.5. DiffPool, TopKPool, and SAGPool are used as hierarchical graph pooling methods in their works, but they achieve good performance as global pooling methods as well [17, 19]. EigenPool from EigenGCN suffers from significant performance loss as a global pooling method  so that we do not include it in the ablation studies. CovPool and AttnPool use the same settings as our proposed methods.
Table II provides the comparison results. Our proposed SOPool and SOPool achieve better performance than DiffPool, SortPool, TopKPool, and SAGPool on all datasets, demonstrating the effectiveness of our graph pooling methods with second-order statistics.
To support our discussion in Section 3.5, we analyze the performance of CovPool and AttnPool. Note that the same bilinear mapping technique used in SOPool is applied on CovPool, in order to avoid the excessive number of parameters. CovPool achieves comparable performance to SOPool on most datasets. However, huge performance loss is observed on PTC, IMDB-BINARY, and IMDB-MULTI, indicating that subtracting the mean is harmful in graph pooling.
Compared to SOPool, AttnPool suffers from performance loss on all datasets except COLLAB and REDDIT-BINARY. The loss is especially significant on bioinformatics datasets (PTC, PROTEINS, NCI1). However, AttnPool achieves the best performance on COLLAB and REDDIT-BINARY among all graph pooling methods, although the added Softmax function results in less discriminative power. The reason might be capturing the distributional information is more important than the exact structure in these datasets. It is similar to GINs, where using averaging as graph pooling achieves better performance on social network datasets than summation .
4.4 Results with Different Graph Neural Networks
We’ve already demonstrated the superiority of our proposed SOPool and SOPool over previous pooling methods. Next, we show that their effectiveness is robust to different GNNs. In this experiment, we change GIN-0 into other six variants of GINs. Note that these variants cover Graph Convolutional Networks (GCN)  and GraphSAGE , thus including a wide range of different kinds of GNNs.
We first give details of different variants of graph isomorphism networks (GINs) . Basically, GINs iteratively update the representation of each node in a graph by aggregating representations of its neighbors, where the iteration is achieved by stacking several layers. Therefore, it suffice to describe the -th layer of GINs based on one node.
Recall that we represent a graph by its adjacency matrix and node feature matrix , where is the number of nodes in and is the dimension of node features. The adjacency matrix tells the neighboring information of each node. We introduce GINs by defining node representation matrices and as inputs and outputs to the -th layer, respectively. We have . Note that the first dimension does not change during the computation, as GINs learn representations for each node.
Specifically, consider a node has corresponding representations and , which are rows of and , respectively. The set of neighboring nodes of is given by . We describe the -layer of the following variants:
Here, the multi-layer perceptron (MLP) has two layers with ReLU activation functions. Note thatMean-1-Layer and Max-1-Layer correspond to GCN  and GraphSAGE , respectively, up to minor architecture modifications.
The results of these different GNNs with our graph pooling methods are reported in Table III. Our proposed SOPool and SOPool achieve satisfying performance consistently. In particular, on social network datasets, the performance does not decline when the GNNs before graph pooling become less powerful, showing the highly discriminative ability of second-order pooling.
4.5 Ablation Studies in Hierarchical Graph Neural Networks
SOPool has shown its effectiveness as global graph pooling through the experiments in Sections 4.2 and 4.3. In this section, we evaluate it as hierarchical graph pooling in hierarchical GNNs. The hierarchical GNN architecture follows the one in , which contains three blocks of a GNN layer followed by graph pooling, as introduced in Section 4.1. The experiments are performed on DD and PROTEINS datasets, where hierarchical GNNs tend to achieve good performance [17, 19].
First, we compare SOPool with different hierarchical graph pooling methods under the same hierarchical GNN architecture. Specifically, we include DiffPool, TopKPool, and SAGPool, which have been used as hierarchical graph pooling methods in their works. The comparison results are provided in Table IV. Our proposed SOPool outperforms all the baselines on both datasets, indicating the effectiveness of SOPool as a hierarchical graph pooling method.
In addition, we conduct experiments to evaluate SOPool in different hierarchical GNNs by varying the number of blocks. The results are shown in Table V. On both datasets, SOPool achieves the best performance when the number of blocks is two. The results indicate current datasets on graph classification are not large enough yet. And without techniques like jumping knowledge networks (JK-Net) , hierarchical GNNs tend to suffer from over-fitting, leading to worse performance than flat GNNs.
In this work, we propose to perform graph representation learning with second-order pooling, by pointing out that second-order pooling can naturally solve the challenges of graph pooling. Second-order pooling is more powerful than existing graph pooling methods, since it is capable of using all node information and collecting second-order statistics that encode feature correlations and topology information. To take advantage of second-order pooling in graph representation learning, we propose two global graph pooling approaches based on second-order pooling; namely, bilinear mapping and attentional second-order pooling. Our proposed methods solve the practical problems incurred by directly using second-order pooling with GNNs. We theoretically show that our proposed methods are more suitable to graph representation learning by comparing with two related pooling methods from computer vision tasks. In addition, we extend one of the proposed method to a hierarchical graph pooling method, which has more flexibility. To demonstrate the effectiveness of our methods, we conduct thorough experiments on graph classification tasks. Our proposed methods have achieved the new state-of-the-art performance on eight out of nine benchmark datasets. Ablation studies are performed to show that our methods outperform existing graph pooling methods significantly and achieve good performance consistently with different GNNs.
This work was supported in part by National Science Foundation grant IIS-1908166, and Defense Advanced Research Projects Agency grant N66001-17-2-4031.
-  P. Yanardag and S. Vishwanathan, “Deep graph kernels,” in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2015, pp. 1365–1374.
-  S. Zhang, H. Tong, J. Xu, and R. Maciejewski, “Graph convolutional networks: Algorithms, applications and open challenges,” in International Conference on Computational Social Networks. Springer, 2018, pp. 79–91.
-  Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu, “A comprehensive survey on graph neural networks,” arXiv preprint arXiv:1901.00596, 2019.
-  W. Fan, Y. Ma, Q. Li, Y. He, E. Zhao, J. Tang, and D. Yin, “Graph neural networks for social recommendation,” in The World Wide Web Conference. ACM, 2019, pp. 417–426.
-  H. Gao, J. Pei, and H. Huang, “Conditional random field enhanced graph convolutional neural networks,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ser. KDD ’19. New York, NY, USA: ACM, 2019, pp. 276–284. [Online]. Available: http://doi.acm.org/10.1145/3292500.3330888
-  J. Ma, P. Cui, K. Kuang, X. Wang, and W. Zhu, “Disentangled graph convolutional networks,” in International Conference on Machine Learning, 2019, pp. 4212–4221.
-  X. Wang, H. Ji, C. Shi, B. Wang, Y. Ye, P. Cui, and P. S. Yu, “Heterogeneous graph attention network,” in The World Wide Web Conference, ser. WWW ’19. New York, NY, USA: ACM, 2019, pp. 2022–2032. [Online]. Available: http://doi.acm.org/10.1145/3308558.3313562
-  T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in International Conference on Learning Representations, 2017.
-  W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in Advances in Neural Information Processing Systems, 2017, pp. 1024–1034.
-  P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” in International Conference on Learning Representations, 2018.
-  K. Xu, C. Li, Y. Tian, T. Sonobe, K.-i. Kawarabayashi, and S. Jegelka, “Representation learning on graphs with jumping knowledge networks,” in International Conference on Machine Learning, 2018, pp. 5449–5458.
-  K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?” in International Conference on Learning Representations, 2019.
-  K. Schütt, P.-J. Kindermans, H. E. S. Felix, S. Chmiela, A. Tkatchenko, and K.-R. Müller, “Schnet: A continuous-filter convolutional neural network for modeling quantum interactions,” in Advances in Neural Information Processing Systems, 2017, pp. 991–1001.
-  D. K. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, and R. P. Adams, “Convolutional networks on graphs for learning molecular fingerprints,” in Advances in Neural Information Processing Systems, 2015, pp. 2224–2232.
-  M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional neural networks on graphs with fast localized spectral filtering,” in Advances in Neural Information Processing Systems, 2016, pp. 3844–3852.
M. Zhang, Z. Cui, M. Neumann, and Y. Chen, “An end-to-end deep learning
architecture for graph classification,” in
Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
-  Z. Ying, J. You, C. Morris, X. Ren, W. Hamilton, and J. Leskovec, “Hierarchical graph representation learning with differentiable pooling,” in Advances in Neural Information Processing Systems, 2018, pp. 4800–4810.
-  H. Gao and S. Ji, “Graph U-Net,” in International Conference on Machine Learning, 2019.
-  J. Lee, I. Lee, and J. Kang, “Self-attention graph pooling,” in International Conference on Machine Learning, 2019, pp. 3734–3743.
-  Y. Ma, S. Wang, C. C. Aggarwal, and J. Tang, “Graph convolutional networks with eigenpooling,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ser. KDD ’19. New York, NY, USA: ACM, 2019, pp. 723–731. [Online]. Available: http://doi.acm.org/10.1145/3292500.3330982
-  Y. Boureau, J. Ponce, and Y. LeCun, “A theoretical analysis of feature pooling in vision algorithms,” in International Conference on Machine learning, vol. 345, 2010.
-  D. Chen, Y. Lin, W. Li, P. Li, J. Zhou, and X. Sun, “Measuring and relieving the over-smoothing problem for graph neural networks from the topological view,” in Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
-  D. G. Lowe, “Object recognition from local scale-invariant features,” in International Conference on Computational Vision, vol. 99, no. 2, 1999, pp. 1150–1157.
-  O. Tuzel, F. Porikli, and P. Meer, “Region covariance: A fast descriptor for detection and classification,” in European conference on computer vision. Springer, 2006, pp. 589–600.
-  ——, “Pedestrian detection via classification on riemannian manifolds,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 10, pp. 1713–1727, 2008.
-  F. Perronnin, J. Sánchez, and T. Mensink, “Improving the fisher kernel for large-scale image classification,” in European Conference on Computer Vision. Springer, 2010, pp. 143–156.
-  J. Carreira, R. Caseiro, J. Batista, and C. Sminchisescu, “Semantic segmentation with second-order pooling,” in European Conference on Computer Vision. Springer, 2012, pp. 430–443.
-  T.-Y. Lin, A. RoyChowdhury, and S. Maji, “Bilinear CNN models for fine-grained visual recognition,” in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1449–1457.
Y. Gao, O. Beijbom, N. Zhang, and T. Darrell, “Compact bilinear pooling,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 317–326.
A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach,
“Multimodal compact bilinear pooling for visual question answering and
visual grounding,” in
Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016, pp. 457–468.
-  Z. Wang and S. Ji, “Learning convolutional text representations for visual question answering,” in Proceedings of the 2018 SIAM International Conference on Data Mining. SIAM, 2018, pp. 594–602.
V. Arsigny, P. Fillard, X. Pennec, and N. Ayache, “Geometric means in a novel vector space structure on symmetric positive-definite matrices,”SIAM Journal on Matrix Analysis and Applications, vol. 29, no. 1, pp. 328–347, 2007.
-  D. Acharya, Z. Huang, D. Pani Paudel, and L. Van Gool, “Covariance pooling for facial expression recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 367–374.
-  Q. Wang, J. Xie, W. Zuo, L. Zhang, and P. Li, “Deep CNNs meet global covariance pooling: Better representation and generalization,” arXiv preprint arXiv:1904.06836, 2019.
-  R. Girdhar and D. Ramanan, “Attentional pooling for action recognition,” in Advances in Neural Information Processing Systems, 2017, pp. 34–45.
-  Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy, “Hierarchical attention networks for document classification,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp. 1480–1489.
-  A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.
-  S. Verma and Z.-L. Zhang, “Graph capsule convolutional neural networks,” arXiv preprint arXiv:1805.08090, 2018.
-  A. K. Debnath, R. L. Lopez de Compadre, G. Debnath, A. J. Shusterman, and C. Hansch, “Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity,” Journal of Medicinal Chemistry, vol. 34, no. 2, pp. 786–797, 1991.
-  H. Toivonen, A. Srinivasan, R. D. King, S. Kramer, and C. Helma, “Statistical evaluation of the predictive toxicology challenge 2000–2001,” Bioinformatics, vol. 19, no. 10, pp. 1183–1193, 2003.
-  K. M. Borgwardt, C. S. Ong, S. Schönauer, S. Vishwanathan, A. J. Smola, and H.-P. Kriegel, “Protein function prediction via graph kernels,” Bioinformatics, vol. 21, no. suppl_1, pp. i47–i56, 2005.
-  N. Wale, I. A. Watson, and G. Karypis, “Comparison of descriptor spaces for chemical compound retrieval and classification,” Knowledge and Information Systems, vol. 14, no. 3, pp. 347–375, 2008.
-  A. Shrivastava and P. Li, “A new space for comparing graphs,” in Proceedings of the 2014 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining. IEEE Press, 2014, pp. 62–71.
-  J. Leskovec, J. Kleinberg, and C. Faloutsos, “Graphs over time: densification laws, shrinking diameters and possible explanations,” in Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery in Data Mining. ACM, 2005, pp. 177–187.
-  P. D. Dobson and A. J. Doig, “Distinguishing enzyme structures from non-enzymes without alignments,” Journal of Molecular Biology, vol. 330, no. 4, pp. 771–783, 2003.
-  M. Niepert, M. Ahmed, and K. Kutzkov, “Learning convolutional neural networks for graphs,” in International Conference on Machine Learning, 2016, pp. 2014–2023.
-  S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International Conference on Machine Learning, 2015, pp. 448–456.
-  N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.
-  D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations, 2015.
-  N. Shervashidze, S. Vishwanathan, T. Petri, K. Mehlhorn, and K. Borgwardt, “Efficient graphlet kernels for large graph comparison,” in Artificial Intelligence and Statistics, 2009, pp. 488–495.
-  S. V. N. Vishwanathan, N. N. Schraudolph, R. Kondor, and K. M. Borgwardt, “Graph kernels,” Journal of Machine Learning Research, vol. 11, no. Apr, pp. 1201–1242, 2010.
-  N. Shervashidze, P. Schweitzer, E. J. v. Leeuwen, K. Mehlhorn, and K. M. Borgwardt, “Weisfeiler-Lehman graph kernels,” Journal of Machine Learning Research, vol. 12, no. Sep, pp. 2539–2561, 2011.
-  S. Ivanov and E. Burnaev, “Anonymous walk embeddings,” in International Conference on Machine Learning, 2018, pp. 2191–2200.
-  J. Atwood and D. Towsley, “Diffusion-convolutional neural networks,” in Advances in Neural Information Processing Systems, 2016, pp. 1993–2001.
-  M. Simonovsky and N. Komodakis, “Dynamic edge-conditioned filters in convolutional neural networks on graphs,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3693–3702.