SStaGCN: Simplified stacking based graph convolutional networks

I Introduction

Rcently, research with graph structural data learning has received wide considerable attention in ample fields of artificial intelligence. Graph neural networks (GNNs), particularly, graph convolutional networks (GCNs)

[1, 2, 3] have shown remarkable success in learning graph structural data, and been applied in recommendation systems [4]

, computer vision

[5], molecular design [6]

, natural language processing

[7], node classification [8] and clustering tasks [9]. Despite their great success, almost all of the GCNs focus on graph data with homogeneous node embedding or bag-of-words representations. Ivanov and Prokhorenkova [10]

incorporated gradient boosted decision trees (GBDT) into GNN and proposed a novel BGNN architecture to deal with heterogeneous tabular data for the first time. As is known to all, there are various types of heterogeneous data in real-world applications. Can we design a general framework to handle distinct types of heterogeneous data besides heterogeneous tabular data. Moreover, as laid out by

[11], graph convolution of GCN model is a special form of Laplacian smoothing, which mixes the node features from different clusters. Therefore, it brings the potential problem of over-smoothing [11]. Sun et al. [12] developed a RNN-like GCN by employing AdaBoost, which can extract knowledge from high-order neighbors of current nodes. However,

A^{l} = B^{l}

does not imply

A = B

, one apparent simple example is considering the situation

l = 2

(two layer), and

A = (\begin{matrix} 0 & 1 1 & 0 \end{matrix}), B = (\begin{matrix} 1 & 0 0 & 1 \end{matrix}) .

Thus, this RNN-like GCN will definitely lose the information of the original graph structure. The over-smoothing, the ability of dealing with heterogeneous data, and the working mechanisms of GCN model remain open.

Recap stacking method [13] or stacked generation is an approach to ensemble multiple different classifications, which contains of base models and meta-model. Stacking is successful in the feature extraction task of tackling different kinds of data, although the data may be disordered or irregular. This is because stacking has the following properties: (1). Combining multiple distinct learners in the base models, effective discernible features could be well learned; (2). Base models are fitted on the whole training data to compute the performance on the test data; (3). Meta-model is used to make a prediction on the test data.

Undoubtedly, we will get benefits by combining stacking and GCN model. To the best of our knowledge, there is no general framework for dealing with heterogenous graph structural data by GCN model. In this paper, we present a novel simplified stacking based architecture for handling graph data, SStaGCN, which combines the feature extraction ability of stacking and aggregation, and GCN’s ability to learn graph structure on graph data. These enable SStaGCN to inherit the advantages of the stacking method and graph convolutional network. Overall, the contributions of the paper are listed as follows:

(1). We propose a new delicate general architecture that combines simplified stacking and GCN, which is adaptive and flexible to tackle heterogeneous graph structural data of different types. To the best of the authors’ knowledge, this is the first systematic work that applies modified stacking approach to general graph structural data.

(2). Generalization bound is addressed to highlight the role of stacking and aggregation from the viewpoint of learning theory.

(3). Extensive evaluation of our approach against strong baselines in node prediction tasks is investigated. The experimental results indicate significant performance improvements on both homogeneous and heterogeneous node classification tasks over a variety of real-world graph structural data, the over-smoothing phenomenon can be well alleviated.

The remainder of the paper is organized as the following. In Section II, we give a brief review of related work. Section III addresses the theoretical analysis and the proposed algorithm for GCN. Experiments on $3$ public citation networks and another $3$ heterogeneous datasets are relegated in Section IV. Some discussions and concluding remarks go to Section V. Proof of the main result is provided in the Appendix.

Ii Related Work

Graph Convolutional Networks

GCNs can be routinely interpreted as extensions of traditional convolutional neural networks on the graph domain. Typically, there are two types of GCNs

[14]

: spatial GCNs and spectral GCNs. Spatial GCNs construct new feature vectors for each vertex using its neighborhood information, in which convolution is viewed as “patch operator”. Spectral GCNs define the convolution by decomposing a graph signal on the spectral domain, and then employing a spectral filter (Fourier or wavelet,

[1], [15], [16]) on the spectral components [17], [18]

. However, this model entails the computation of the Laplacian eigenvector, which is exhausted and impractical for large-scale graphs. Hammond et al.

[19] used Chebyshev polynomials up to

K -

th order to approximate the spectral filter. Defferrard and Vandergheynst [20] constructed a

K

-localized ChebyNet. Kipf and Welling [8] considered the case

K = 1

and proposed a simple but powerful model for semi-supervised classification task. Wu et al. [21] removed nonlinearities and collapsed the weight matrix between consecutive layers, achieved a simplified GCN, whilst [22, 23] considered the design of deep GCNs. Multi-scale deep GCNs were investigated in [24].

Ensemble learning based graph neural networks

GCNs may confront with the over-smoothing problem and can not handle heterogeneous graph data. Sun et al. [12] designed a RNN-like graph structure to extract the knowledge from high-order neighbors of the current nodes, while Ivanov and Prokhorenkova [10] incorporated GBDT into GNN and developed BGNN to tackle heterogeneous tabular data. A natural question arises: Is there a general GCN architecture to handle various heterogeneous graph structural data and mitigate the over-smoothing issue? This paper aims at investigating these challenges and answers the above-mentioned questions.

Iii The proposed approach: SStaGCN

Iii-a Graph convolutional networks

Given an undirected graph $G = (V, E)$ with $N$ nodes $v_{i} \in V$ , edges $(v_{i}, v_{j}) \in E$ . Denote $A \in R^{N \times N}$ as the adjacency matrix with corresponding degree matrix $D_{i i} = \sum_{j} A_{i j}$ . Obviously, for an undirected graph, $A_{i j} = A_{j i}$ . In the conventional GCN models for semi-supervised node classification task, the graph embedding of nodes with two convolutional layers is described as the following:

Z =^A R e L U (^A X W^{(0)}) W^{(1)}

(1)

where $Z \in R^{N \times K}$

is the final embedding matrix (output logits) of nodes before softmax with

K

the number of classes.

X \in R^{N \times d}

stands for the feature matrix with

d

the input dimension.

^A = {~ D}^{- \frac{1}{2}} ~ A {~ D}^{- \frac{1}{2}}

, where

~ D

is the degree matrix of

~ A

and

~ A = A + I

(

I

stands for identity matrix). Moreover,

W^{(0)} \in R^{d \times H}

is the input-to-hidden weight matrix for a hidden layer with

H

feature maps, and

W^{(1)} \in R^{H \times K}

denotes the hidden-to-output weight matrix.

Iii-B Stacking

Stacking, as a hierarchical model integration framework, is a well-known and widely used ensemble machine learning algorithm

[25]

. It uses a meta-learning algorithm to learn how to best combine the prediction from two or more base machine learning algorithms. Traditional stacking model involves two or more base models, and a meta-model that combines the predictions of the base models. Base models use different types of models to fit on the training data and compile the predictions. Meta-model tries to best combine the predictions of the base models, which is often simple, providing a smooth interpretation of the predictions made by the base models. Hence, linear models are often used as the meta-model, such as linear regression for regression tasks and logistic regression for classification tasks.

Iii-C The proposed approach

As mentioned above, GCN may mix the node features from different clusters and make them indistinguishable. Therefore, it is necessary to aggregate more node information in an effective way for better predictions. Motivated by traditional stacking approach and the work in [12, 10], to reduce the computational cost, we only use base models of the stacking approach, and then aggregate the output of them to attain the node features of the graph data. Specifically, the proposed method could be addressed as follows. At first, in the first layer, we attain $X^{'}$ by input $X \in R^{N \times d}$ (node feature matrix) through $k$

base classifiers

X_{i}^{'} = h_{i} (X), i = 1, \dots, k,

(2)

Thereafter, we get the pre-classification results $X_{i}^{'}$ through $h_{i} (X)$ ( $i = 1, \dots, k$ ). Secondly, we use aggregation method to attain the final output results, i. e.,

~ X = g (X_{1}^{'}, \dots, X_{k}^{'}),

(3)

where $g (\cdot)$ denotes an aggregation method. The idea of aggregation method is very simple, which aims to group attribute values by a single value. Generally, we can choose mean, attention or voting approach. Recall

0: Feature matrix

X

, normalized adjacency matrix

^A

, graph

G

, base classifier

h_{i} (X)

i = 1, \dots, k

, aggregation method

g (\cdot)

;

0: Final predictor

f (X)

;

1: Attain

X_{i}^{'}

(

i = 1, \dots, k

) via

k

base classifiers;

X_{i}^{'} \leftarrow h_{i} (X)

i = 1, \dots, k

;

2: Aggregate

X_{1}^{'}, \dots, X_{k}^{'}

;

~ X \leftarrow g (X_{1}^{'}, \dots, X_{k}^{'})

;

3: Feed

~ X

into vanilla GCN ;

f (X) \leftarrow G C N (~ X)

;

4: return

f (X)

;

Algorithm 1 SStaGCN.

Mean: mean operator takes the element-wise mean of the components $X_{1}^{'}, \dots, X_{k}^{'}$ .

Attention[26]

: Attention mechanism has been widely used in various fields of deep learning, including but not limited to image processing

[27], speech recognition[28], and natural language processing[29]. The idea of attention is motivated from the attention mechanism of human beings. Denote query vector

V

as the output of base classifiers, let query vector

Y

be the label of the data. Then we compute the attention coefficients between

Y

and

V

as follows:

a_{i} = s o f t m a x (c o s (Y, V_{i})), i = 1, \dots, k,

(4)

where $cos$

denotes cosine similarity. Finally, the input

~ X

of the graph convolutional layer is aggregated by considering the following sum with attention score

a_{i}

~ X = k \sum i = 1 a_{i} V_{i} .

(5)

Voting [30]: Voting method is the most intuitive way among all integrated learning methods, which aims at selecting one or more winners. In this work, our objective is to choose the most popular prediction among the results of the base classifiers. Therefore, we will take the majority voting approach. if the number of categories appears the same, a category will be randomly selected.

Thus, we attain a novel GCN model to deal with heterogeneous graph data by elegantly combining stacking, aggregation, and vanilla GCN, namely, SStaGCN. The first layer of SStaGCN utilizes the base models of stacking approach, and the second layer of SStaGCN uses aggregation method such as mean, attention or voting, which can enhance the feature extraction ability of conventional GCN models. The aggregated data will henceforth be used as the input of conventional GCN model, and we attain the final prediction results. The workflow of the proposed model is demonstrated in Algorithm 1 and Fig. 1.

Iii-D Generalization Bound Analysis

In this Section, we give theoretical generalization analysis of the proposed approach. In our analysis, we assume that the adjacency matrix $A$ and the node feature matrix $X$ are both fixed.

Theoretically, in learning theory, the risk of $f$ over the unknown population distribution $P$ is measured by

E (f) := E_{X \times Y} [ℓ (y, f (~ x))],

where $ℓ$

is the loss function defined as a map:

ℓ : X \times Y \to R^{+}

, Given a training data and adjacency matrix

A

, the objective is to estimate parameters

(W^{(0)}, W^{(1)})

from model (1) based on empirical data. Concretely, we attempts to minimize the empirical risk functional over some function class

F

takes form

E_{m} (f) := \frac{1}{m} m \sum j = 1 ℓ (y_{j}, f ({~ x}_{j})),

where ${({~ x}_{j}, y_{j})_{j = 1}^{m}}$ is the labelled sample achieved from the original training data ${(x_{j}, y_{j})_{j = 1}^{m}}$ via stacking and aggregation. Typically, the clustering algorithms will only produce discernible nodes. Hence, if $∥ x_{i} ∥ \leq R, i = 1, \dots, N$ . It is trivial that stacking and the three aggregation methods proposed in this paper will not violate the constraints, i.e., $∥ {~ x}_{i} ∥ \leq R, i = 1, \dots, N$ . Now we are in position to present the theoretical generalization bound analysis.

Theorem 1

Suppose $∥ x_{i} ∥_{2} \leq R, i = 1, \dots, N$ , $∥ W^{(0)} ∥_{F} \leq B_{1}$ , $∥ W^{(1)} ∥_{F} \leq B_{2}$ . Denote $N (i)$ the number of neighbors of node $v_{i} \in Ω$ (the set of node indices with observed labels), let $q = max {N (i)}$ , $f : X \to R$ be any given predictor of a class of GCNs with one-hidden layer. Assume that the loss function $ℓ (y, \cdot)$ is Lipschitz continuous with Lipschitz constant $α_{ℓ}$ . Then, for any $δ > 0$

, with probability at least

1 - δ

, we have

	$E (f) \leq E_{N} (f)$	$+ \frac{2 α_{ℓ} \sqrt{2 q} K B_{1} B_{2} R \sum_{s = 1}^{q} M_{s}}{\sqrt{m}}$
		$+ \sqrt{\frac{2 log (2 / δ)}{N}},$

where $∥ \cdot ∥_{F}$ is the Frobenius norm, $M_{s} = {max}_{i \in [N]} | {^A}_{i v} |$ with $v \in N (i)$ .

Remark 1

Theorem 1 indicates that the dominant upper bound linearly depends on the maximum number of the neighbors of the nodes $q$ , bounds of the weights $B_{1}$ and $B_{2}$ , which strongly depend on the dimension index $d_{1}$ . Obviously, $d_{1} ≪ d$ will yield a tight generalization bound. When $K = 1$ (binary classification case), the results stated here is similar to the one outlined in [31].

Iv Experiments

Iv-a Datasets

To evaluate the performance of the proposed SStaGCN model for distinct types of graph structural data, we utilize $6$ real-world datasets for the semi-supervised node classification task, including $3$

commonly used citation networks: Cora, CiteSeer, and Pubmed

[32], and another

3

heterogeneous datasets: House

_

class, VK

_

class and DBLP [10]. As indicated in [10], House

_

class and VK

_

class are from House and VK datasets, respectively, where the target labels are converted into several discrete classes due to the lack of publicly available heterogeneous graph-structured data.

In the $3$ citation networks, nodes represent documens, and edges (undirected) stand for the citation relationships connected to documents. The characteristics of nodes are representative words in documents, and the label rate here denotes the percentage of node tags used for training. The Cora dataset contains $2708$ nodes, $5429$ edges, $7$ classes, and $1433$ node features, the CiteSeer dataset contains $3327$ nodes, $4732$ edges, $6$ classes, and $3703$

node features, and the Pubmed dataset contains

19, 717

nodes,

44, 338

edges, and

3

classes. We select

140

120

, and

60

nodes for Cora, CiteSeer, and Pubmed datasets for training, respectively. Each dataset uses

1000

nodes for testing, and

500

nodes for cross-validation. The data splitting we used is the same as that of GCN, Graph Attention Network (GAT, [33]) and GWNN [15]. Details about the citation network (heterogeneous network resp.) are described in Table I (Table II resp.)

Dataset	Cora	CiteSeer	Pubmed
Nodes	2708	3327	19717
Edges	5429	4732	44338
Features	1433	3703	500
Classes	7	6	3
Label Rate	5.2%	3.6%	0.3%

TABLE I: Summary of the citation networks.

Dataset	House $_$ class	VK $_$ class	DBLP
Nodes	20640	54028	14475
Edges	182146	213644	40269
Features	6	14	5002
Classes	5	7	4
Min Target Nodes	0.14	13.48	745
Max Target Nodes	5.00	118.39	1197

TABLE II: Summary of the heterogeneous data.

Iv-B Baselines

We compare SStaGCN with $4$ classical graph convolutional networks: ChebyNet, GCN, GAT, and APPNP [34], and $2$ ensemble learning based GCNs: AdaGCN and BGNN models.

Iv-C Setting

As demonstrated in Fig. 1, the proposed SStaGCN model contains four layers, where the first and second layers are called feature extraction layer. The first layer is based on the base models of the stacking method. Here we consider the combination of $7$

classical classifiers: KNN, Random Forest, Naive Bayes, Decision Tree, SVC

[35], GBDT [36], and Adaboost[37] in the first layer of SStaGCN. These

7

classifiers are representative classical classifiers used in the community of machine learning, which have respective merits in dealing with distinct types of tasks. Moreover, we adopt three aggregation methods: mean, attention, and voting in the second layer. Among them, for the mean approach, we take the mean value of the output of the first layer and then round it. As for the attention mechanism, we consider the label data as vector

Y

, the predicted value of the feature extraction layer as the query vector

V

, and then compute the attention coefficients. For the voting approach, we employ the hard voting technique in ensemble learning [38].

Thereafter, the output of the second layer is considered as the input of the first layer of the GCN, which only has two layers in our setting. The GCN considered in this paper has $16$ hidden units, and Adam optimizer [39] is the default optimizer, cross entropy is used as the loss function. We set learning rate $γ = 0.01$ , number of iterations $i t r = 500$ , weight decay $5 e - 4$ , and dropout rate equals $0$ .

Iv-D Results

Method	Cora	CiteSeer	Pubmed
ChebyNet	81.20 $\pm 0.00$	69.80 $\pm$ 0.00	74.40 $\pm$ 0.00
GCN	81.50 $\pm$ 0.00	70.30 $\pm$ 0.00	79.00 $\pm$ 0.00
GAT	83.00 $\pm$ 0.70	72.50 $\pm$ 0.70	79.00 $\pm$ 0.30
APPNP	85.09 $\pm$ 0.25	75.73 $\pm$ 0.30	79.73 $\pm$ 0.31
AdaGCN	85.97 $\pm$ 0.20	76.68 $\pm$ 0.20	79.95 $\pm$ 0.21
BGNN	41.97 $\pm$ 0.19	30.74 $\pm$ 0.10	10.32 $\pm$ 0.10
Sim $_$ Stacking	43.19 $\pm$ 2.05	62.7 $\pm$ 0.53	87.7 $\pm$ 0.23
SStaGCN (Mean)	90.35 $\pm$ 0.20	86.40 $\pm$ 0.12	82.30 $\pm$ 0.19
SStaGCN (Attention)	91.60 $\pm$ 0.18	87.20 $\pm$ 0.12	82.40 $\pm$ 0.23
SStaGCN (Voting)	93.10 $\pm$ 0.16	88.70 $\pm$ 0.14	92.07 $\pm$ 0.20

TABLE III: Average accuracy on

3

citation networks under

30

runs by computing the

95 %

confidence interval via bootstrap

Method	House $_$ class	VK $_$ class	DBLP
ChebyNet	54.74 $\pm$ 0.10	57.19 $\pm$ 0.36	32.14 $\pm$ 0.00
GCN	55.07 $\pm$ 0.13	56.40 $\pm$ 0.09	39.49 $\pm$ 1.37
GAT	56.50 $\pm$ 0.22	56.42 $\pm$ 0.19	76.83 $\pm$ 0.78
APPNP	57.03 $\pm$ 0.27	56.72 $\pm$ 0.11	79.47 $\pm$ 1.46
AdaGCN	26.20 $\pm$ 0.00	46.00 $\pm$ 0.00	10.06 $\pm$ 0.00
BGNN	66.7 $\pm$ 0.27	66.32 $\pm$ 0.20	86.94 $\pm$ 0.74
Sim $_$ Stacking	53.89 $\pm$ 0.29	56.64 $\pm$ 0.10	71.58 $\pm$ 0.64
SStaGCN (Mean)	72.35 $\pm$ 0.05	66.62 $\pm$ 0.17	82.31 $\pm$ 0.20
SStaGCN (Attention)	72.40 $\pm$ 0.12	77.64 $\pm$ 0.08	82.51 $\pm$ 0.22
SStaGCN (Voting)	76.13 $\pm$ 0.12	87.92 $\pm$ 0.07	92.60 $\pm$ 0.10

TABLE IV: Average accuracy on heterogeneous datasets under

30

runs by calculating the

95 %

confidence interval via bootstrap

Method	Cora	CiteSeer	Pubmed
ChebyNet	77.99 $\pm$ 0.54	63.76 $\pm$ 0.34	77.74 $\pm$ 0.42
GCN	82.89 $\pm$ 0.30	70.65 $\pm$ 0.37	78.83 $\pm$ 0.32
GAT	83.59 $\pm$ 0.25	70.62 $\pm$ 0.29	77.77 $\pm$ 0.40
APPNP	84.29 $\pm$ 0.22	71.05 $\pm$ 0.38	79.66 $\pm$ 0.31
AdaGCN	79.55 $\pm$ 0.19	63.62 $\pm$ 0.19	78.55 $\pm$ 0.21
BGNN	40.81 $\pm$ 0.25	32.73 $\pm$ 0.13	8.46 $\pm$ 0.08
Sim $_$ Stacking	44.02 $\pm$ 1.61	60.86 $\pm$ 0.56	87.31 $\pm$ 0.13
SStaGCN (Mean)	90.66 $\pm$ 0.18	86.42 $\pm$ 0.12	82.30 $\pm$ 0.19
SStaGCN (Attention)	91.69 $\pm$ 0.14	87.24 $\pm$ 0.14	82.45 $\pm$ 0.23
SStaGCN (Voting)	92.76 $\pm$ 0.16	88.73 $\pm$ 0.14	92.07 $\pm$ 0.20

TABLE V: Average F1-score (macro) on

3

citation networks under

30

runs by computing the

95 %

confidence interval via bootstrap

Method	House $_$ class	VK $_$ class	DBLP
ChebyNet	31.34 $\pm$ 0.12	57.44 $\pm$ 0.27	26.84 $\pm$ 0.62
GCN	54.95 $\pm$ 0.14	56.52 $\pm$ 0.09	38.5 $\pm$ 0.97
GAT	56.54 $\pm$ 0.68	56.41 $\pm$ 0.07	77.1 $\pm$ 1.86
APPNP	57.88 $\pm$ 0.32	56.61 $\pm$ 0.07	79.34 $\pm$ 0.23
AdaGCN	25.01 $\pm$ 0.00	37.03 $\pm$ 0.00	9.60 $\pm$ 0.00
BGNN	66.48 $\pm$ 0.22	66.18 $\pm$ 0.11	87.2 $\pm$ 0.60
Sim $_$ Stacking	53.32 $\pm$ 0.15	56.11 $\pm$ 0.08	71.49 $\pm$ 0.31
SStaGCN (Mean)	72.23 $\pm$ 0.04	66.74 $\pm$ 0.21	82.13 $\pm$ 0.38
SStaGCN (Attention)	72.36 $\pm$ 0.09	77.62 $\pm$ 0.10	82.68 $\pm$ 0.12
SStaGCN (Voting)	75.45 $\pm$ 0.82	87.84 $\pm$ 0.04	92.64 $\pm$ 0.06

TABLE VI: Average F1-score(macro) on heterogeneous datasets under

30

runs by calculating the

95 %

confidence interval via bootstrap

Models	Cora	CiteSeer	Pubmed	House $_$ class	VK $_$ class	DBLP
ChebyNet	2.59e-06	2.77e-08	1.17e-06	3.51e-10	1.11e-11	6.97e-09
GCN	4.19e-16	5.15e-17	1.84e-15	2.36e-08	2.25e-11	2.48e-07
GAT	2.10e-19	1.54e-20	8.84e-19	3.35e-08	1.38e-09	4.95e-06
APPNP	6.86e-42	7.12e-42	6.25e-41	7.05e-15	1.39e-21	2.26e-09
AdaGCN	1.82e-19	1.23e-20	8.42e-19	3.05e-38	7.94e-43	1.69e-35
BGNN	1.02e-24	2.33e-21	4.74e-22	6.54e-07	1.07e-08	0.11e-03
Sim $_$ Stacking	5.61e-12	3.79e-15	2.52e-09	4.56e-08	6.07e-11	1.07e-06

TABLE VII:

p-values of the paired t-test of SStaGCN (Voting) with competitors on

6

different data sets (CORA, Citeseer, Pubmed, House

_

class, VK

_

class, and DBLP).

The results of the comparative evaluation for node classification are summarized in Tables III-XI, where SStaGCN (Mean) stands for mean mechanism is utilized in the second layer of SStaGCN, SStaGCN (Attention) and SStaGCN (Voting) have similar meanings. We report the accuracy, F1-score (macro), and training time on the test set between the proposed SStaGCN model and other methods. Experimental results successfully demonstrate significant improvement of SStaGCN model over the baselines. Specifically, for $3$ public citation networks, SStaGCN (Voting) achieves almost $8 %$ ( $8 %$ ), $12 %$ ( $17 %$ ), and $13 %$ ( $12 %$ ) improvement of the accuracy (resp. F1-score) for Cora, CiteSeer and Pubmed datasets, respectively. For heterogeneous datasets, SStaGCN (Voting) obtains nearly $9 %$ ( $9 %$ ), $21 %$ ( $21 %$ ), and $6 %$ ( $5 %$ ) improvement of the accuracy (resp. F1-score) for House $_$ class, VK $_$ class, and DBLP datasets, respectively. AdaGCN performs bad on heterogeneous dataset. This maybe due to the reason that AdaGCN aims at designing deep GCNs, which may mix the node features of different clusters when the layers of GCNs go deeper. Intuitively, SStaGCN is able to enhance the performance of GCN, and provides better qualitative results for distinct types of graph structural data.

This impressive improvement can be explained as follows:

(1). The feature extraction step of SStaGCN can achieve a dimensionality reduction effect and make the graph data more discernible. For instance, the size of Cora dataset reduces to $2708 \times 7$ from $2708 \times 1433$ after conducting feature extraction, which means we attain a relatively smaller $d_{1}$ as discussed in Remark 1, and this greatly improves the prediction ability and computation efficiency in the subsequent graph convolution model.

(2). In the aggregation step of SStaGCN model, the mean and attention mechanisms destroy the pre-classification results to some extent, which is inappropriate for feature extraction, while voting mechanism does not. Therefore, experimental results demonstrate that SStaGCN (Voting) is more efficient in our datasets.

(3). Simplified stacking can extract efficient node features but ignore the graph structure information, while GCN model is weak at extracting node features. Hence, the SStaGCN model inherits the merits of simplified stacking and GCN, not only achieves higher classification accuracy, but also reduces the cost of computation time.

Tables VIII and IX indicate the comparison of training time between SStaGCN and other methods. We can see that BGNN runs faster on $3$ citation networks followed by our SStaGCN method, whilst the proposed SStaGCN runs faster on heterogeneous datasets except on DBLP dataset. We think the reason is that the feature extraction step takes extra computation time but yields more efficiency when the output of features is fed into the GCN model.

Method	Cora	CiteSeer	Pubmed
ChebyNet	22.74 $\pm$ 1.24	30.87 $\pm$ 1.21	124.99 $\pm$ 1.77
GCN	13.41 $\pm$ 0.16	99.21 $\pm$ 0.98	55.61 $\pm$ 0.73
GAT	20.98 $\pm$ 0.46	30.74 $\pm$ 1.40	126.33 $\pm$ 1.80
APPNP	203.75 $\pm$ 0.15	55.40 $\pm$ 0.40	457.62 $\pm$ 12.77
AdaGCN	772.26 $\pm$ 83.56	2129.02 $\pm$ 148.97	2098.10 $\pm$ 275.88
BGNN	1.33 $\pm$ 0.00	2.40 $\pm$ 0.00	2.54 $\pm$ 0.00
Sim $_$ Stacking	11.9 $\pm$ 0.08	27.9 $\pm$ 0.24	79.1 $\pm$ 1.40
SStaGCN (Mean)	10.9 $\pm$ 0.13	17.2 $\pm$ 0.13	89.6 $\pm$ 2.20
SStaGCN (Attention)	11.2 $\pm$ 0.24	17.6 $\pm$ 0.41	87.6 $\pm$ 0.96
SStaGCN (Voting)	16.2 $\pm$ 0.19	29.6 $\pm$ 1.40	13.1 $\pm$ 2.61

TABLE VIII: Average Training time(s) on

3

citation networks by computing the

95 %

confidence interval via bootstrap

Method	House $_$ class	VK $_$ class	DBLP
ChebyNet	833.82 $\pm$ 0.00	1394.68 $\pm$ 0.00	8890.74 $\pm$ 0.00
GCN	46.06 $\pm$ 0.80	120.1 $\pm$ 3.35	268.5 $\pm$ 5.35
GAT	197.6 $\pm$ 3.08	410.9 $\pm$ 9.95	205.3 $\pm$ 2.00
APPNP	129.7 $\pm$ 3.26	383.8 $\pm$ 12.02	176.8 $\pm$ 5.58
AdaGCN	607.31 $\pm$ 0.00	511.41 $\pm$ 0.00	590.90 $\pm$ 0.00
BGNN	26.37 $\pm$ 1.65	93.47 $\pm$ 6.23	50.25 $\pm$ 3.04
Sim $_$ Stacking	16.41 $\pm$ 0.36	43.58 $\pm$ 2.78	380.8 $\pm$ 12.82
SStaGCN (Mean)	52.94 $\pm$ 0.51	132.9 $\pm$ 1.61	188.7 $\pm$ 0.54
SStaGCN (Attention)	57.38 $\pm$ 1.75	133.2 $\pm$ 1.11	246.2 $\pm$ 0.37
SStaGCN (Voting)	59.69 $\pm$ 0.82	154.1 $\pm$ 1.02	310.7 $\pm$ 0.49

TABLE IX: Average Training time(s) on heterogeneous datasets by calculating the

95 %

confidence interval via bootstrap

To express the effect of feature extraction step of the proposed model, we provide a visualization with t-SNE [40] as shown in Figs. 2(a) and 3(a). Figs. 2(a) and 3(a) indicate that the combination of stacking and aggregation could well extract the node features and make the graph data more discriminative. Moreover, the paired t-test in Table VII demonstrates that the proposed SStaGCN model is significantly different from the simplified stacking and other GCN models.

Table X indicates that we do not need all the seven classifiers. Taking Cora dataset as an example, we can observe that KNN, Random Forest, and Naive Bayes is the best combination, which attains the highest accuracy value without much cost in computation time (only $3$ seconds). Therefore, this demonstrates that the $7$ classifiers have their own merits in handling different specific tasks.

KNN	Random Forest	Naive Bayes	Decision Tree	GBDT	Adaboost	SVC	Accuracy	Training Time
	$\sqrt$	$\sqrt$					91.2	13.90
$\sqrt$	$\sqrt$	$\sqrt$					93.6	16.60
				$\sqrt$	$\sqrt$		84.2	567.9
		$\sqrt$	$\sqrt$			$\sqrt$	92.9	144.7
	$\sqrt$	$\sqrt$				$\sqrt$	93.1	149.5
	$\sqrt$	$\sqrt$	$\sqrt$				92.8	15.90
$\sqrt$	$\sqrt$	$\sqrt$	$\sqrt$				93.4	18.80
$\sqrt$	$\sqrt$	$\sqrt$	$\sqrt$	$\sqrt$			92.9	568.3
$\sqrt$	$\sqrt$	$\sqrt$	$\sqrt$	$\sqrt$	$\sqrt$		92.9	570.7
$\sqrt$	$\sqrt$	$\sqrt$	$\sqrt$	$\sqrt$	$\sqrt$	$\sqrt$	92.8	708.5

TABLE X: Accuracy and training time(s) on Cora dataset by combinations of different classifiers based on SStaGCN model.

Furthermore, to demonstrate the effect of simplified stacking to the over-smoothing problem, we also add an experiment on the over-smoothing discussion. In Fig. 4(a), we can observe that conventional GCN may mix the features of vertices from different clusters when increasing the layers of GCN. However, as demonstrated in Fig. 5(a) and Table XI ¹¹1here the number of layers do not contain the number of layers included in feature extraction part (only $2$ layers) of SStaGCN., the proposed SStaGCN could effectively ameliorate the over-smoothing phenomenon and improve the accuracy.

Method	2-layer	3-layer	4-layer	5-layer	6-layer	7-layer
GCN	80.5	80.4	75.8	71.9	72.6	60.8
SStaGCN	93.3	88.8	87.5	86.4	84.8	84.3

TABLE XI: Accuracy comparison between GCN and SStaGCN models on Cora dataset using distinct number of layers.

Overall, these experiments demonstrate the superiority of SStaGCN model over competitors.

Iv-E Visualization

To further demonstrate the performance of SStaGCN, we plot the final classification features via GCN, AdaGCN, BGNN, and our SStaGCN. Fig. 6(a) (Fig. 7(a) resp.) displays the final classification features of relevant methods on CiteSeer (DBLP resp.) dataset. From Figs. 6(a) and 7(a), we can observe that relatively smaller points are misclassified by the proposed SStaGCN, whilst many classes are wrongly predicted and classified by GCN, AdaGCN and BGNN.

V Conclusion

Traditional GCNs could not well deal with heterogeneous graph structural data. In this work, we propose a novel GCN architecture, namely, SStaGCN. SStaGCN first takes advantages of stacking method and aggregation technique to attain pre-classified data features, and then utilizes GCN to conduct prediction for heterogeneous graph data. Our approach SStaGCN can effectively explore and exploit node features for heterogeneous graph data in a stacking way. Our work paves a way towards better combining classical machine learning methods to design GCN models, proposes a general framework for handling distinct types of graph structural data, and will definitely give insights to a better understanding of GCN. Extensive experiments demonstrate that the proposed model is superior to state-of-the-art competitors in terms of accuracy, F1-score, and training time. The proposed framework here could be generalized to the regression setting. Furthermore, we believe the proposed method can tackle various distinct types of heterogeneous graph data, although experiments are conducted on tabular data. A promising future research direction is to investigate deeper GCNs in our setting as discussed by [12].

Vi Appendix

In this part, we give the detailed proof of Theorem 1. Before addressing the proof, we give several lemmas. At first, let us present the contraction inequality of Rademacher complexity in the vector form.

Lemma 1

[41] Let $X$ be any set, $(x_{1}, \dots, x_{N}) \in X$ and $F$ be a class of functions $f : X \to R^{K}$ ad let $h_{i} : R^{K} \to R$ have Lipschitz constant $L$ . Then

E sup f \in F N \sum i = 1 σ_{i} h_{i} f (x_{i}) \leq \sqrt{2} L E sup f \in F N \sum i = 1 K \sum k = 1 σ_{i k} f_{k} (x_{i}),

where $σ_{i k}$ is an independent doubly indexed Rademacher sequence and $f_{k} (x_{i})$ is the $k -$ th component of $f (x_{i})$ .

Lemma 2

[42] Consider a loss function $ℓ : X \times Y \to R^{+}$ . Denote $H = {ℓ (y, f (\cdot)), f \in F}$ , and let $(x_{i}, y_{i})_{i = 1}^{N}$ be independently selected according to the probability measure $P$ . Then for any $0 < δ < 1$ , with probability at least $1 - δ$ ,

E (f) \leq E_{N} (f) + 2 ˆ R (H) + \sqrt{\frac{2 log (2 / δ)}{N}}, \forall f \in H .

We first give a lemma which plays essential role in the proof of Theorem 1.

Lemma 3

Let $q = max {N (i)}$ for each node $v_{i} \in Ω$ , then

∥ \sum j \in N (v) {^A}_{v j} x_{j} ∥_{2}^{2} \leq R^{2} q .

Proof 1

Denote ${^A}_{v} \in R^{q \times q}$ as the sub-matrix of $^A$ whose row and column indices belong to the set $j \in N (v)$ . Let ${~ X}_{v} = (x_{1}^{T}, \dots, x_{q}^{T})^{T} \in R^{q \times d}$ be the feature matrix of the nodes in $G_{v}$ (subgraph of $G$ ). Hence,

∥ ∥ \sum j \in N (v) {^A}_{v j} x_{j} {∥ ∥}_{2}^{2} = ∥ {^A}_{v \cdot} {~ X}_{v} ∥_{2}^{2} \leq ∥ {^A}_{v \cdot} ∥_{2}^{2} ∥ {~ X}_{v} ∥_{2}^{2},

where ${^A}_{v \cdot}$ is the $v -$ th row of the matrix $^A$ with column index $j$ belong to the set $N (v)$ . Notice that

∥ {~ X}_{v} ∥_{2} = sup ∥ t ∥_{2} = 1 ∥ {~ X}_{v} t ∥_{2} \leq \sqrt{q \sum s = 1 ∥ x_{s} ∥_{2}^{2}} \leq R \sqrt{q},

and $∥^A ∥_{2} \leq 1$ . Therefore,

∥ ∥ \sum j \in N (v) {^A}_{v j} x_{j} {∥ ∥}_{2}^{2} \leq ∥^A ∥_{2}^{2} ∥ {~ X}_{v} ∥_{2}^{2} \leq R^{2} q .

Now we are in position to give the proof of Theorem 1.

Proof 2

To allow a slight abuse of notations, we will use $x_{j}$ to denote ${~ x}_{j}$ due to the explanation on page 3. Denote $h (W^{(0)}) = R e L U (^A X W^{(0)})$ , $f (W^{(1)}) = s o f t m a x (^h (W^{(0)}) W^{(1)}) = (f_{1} (W^{(1)}), \dots, f_{m} (W^{(1)}))^{T}$ with $^h (W^{(0)}) =^A h (W^{(0)})$ . Applying Proposition 4 in [43] to the case $λ = 1$ , we can attain the Lipschitz constant for standard softmax function is $L = 1$ . Let ${^A}_{i \cdot}$ stands for the $i -$ th row of the matrix $^A$ , for function set

	$FB1,B2={fi=softmax(^Ai⋅ReLU(^AXW(0))W(1)),$
	$i = 1, \dots, m, ∥ W^{(0)} ∥_{F} \leq B_{1}, ∥ W^{(1)} ∥_{F} \leq B_{2}},$

the empirical Rademacher complexity is defined as

^R (F_{B_{1}, B_{2}}) = E_{σ} [\frac{1}{} m sup ∥ W^{(0)} ∥_{F} \leq B_{1} ∥ W^{(1)} ∥_{F} \leq B_{2} m \sum i = 1 σ_{i} f_{i} (W^{(1)})],

where ${σ_{i}}_{i = 1}^{m}$ is an i.i.d. family of Rademacher variables independent of $x_{i}$ . By the contraction property of Rademacher complexity,

^R (H) \leq α_{ℓ}^R (F_{B_{1}, B_{2}}),

and notice Lemma 2, we only need to bound $^R (F_{B_{1}, B_{2}})$ . Therefore, we have the following estimate by utilizing Lemma 1.

where $W^{(1)} = (w_{1}^{(1)}, \dots, w_{K}^{(1)})$ , notice the property of inner product, the above estimate can be further bounded as

	$^R (F_{B_{1}, B_{2}})$


	$\leq \frac{\sqrt{2} B_{2}}{m} E_{σ} [sup ∥ W^{(0)} ∥_{F} \leq B_{1} K \sum k = 1 ∥ ∥ m \sum i = 1 σ_{i k} {^h}_{i} (W^{(0)}) {∥ ∥}_{2}],$
	$\leq \frac{\sqrt{2} B_{2}}{m} E_{σ} [K \sum k = 1 sup ∥ W^{(0)} ∥_{F} \leq B_{1} ∥ ∥ m \sum i = 1 σ_{i k} {^h}_{i} (W^{(0)}) {∥ ∥}_{2}],$

the last inequality follows by the property that $sup (\sum_{s} a_{s}) \leq \sum_{s} sup (a_{s})$ . Now the key point is how to estimate the term ${sup}_{∥ W^{(0)} ∥_{F} \leq B_{1}} ∥ ∥ \sum_{i = 1}^{m} σ_{i k} {^h}_{i} (W^{(0)}) {∥ ∥}_{2}$ . We will employ the idea introduced in [31] (in the proof of Theorem 1) to remove the “sup” term. Let $h_{v} (W^{(0)})$ $=$ $(h_{v}^{1} (w_{1}^{(0)})$ , $h_{v}^{2} (w_{2}^{(0)})$ , $\dots$ , $h_{v}^{H} (w_{H}^{(0)}))$ with $W^{(0)}$ $=$ $(w_{1}^{(0)}$ , $\dots$ , $w_{H}^{(0)})$ and notice that $^h (W^{(0)}) =^A h (W^{(0)})$ , then we have

	$∥ ∥ m \sum i = 1 σ_{i k} {^h}_{i} (W^{(0)}) {∥ ∥}_{2}^{2}$
	$= H \sum t = 1 (m \sum i = 1 σ_{i k} \sum v \in N (i) {^A}_{i v} h_{v}^{t} (w_{t}^{(0)}))_{2}^{2}$
	$= H \sum t = 1 ∥ w_{t}^{(0)} ∥_{2}^{2} (m \sum i = 1 σ_{i k} \sum v \in N (i) {^A}_{i v} h_{v}^{t} (w_{t}^{0} / ∥ w_{t}^{(0)} ∥_{2}))^{2} .$

By the definition of Frobenius norm $∥ W ∥_{F}^{2} = \sum_{t = 1}^{H} ∥ w_{t} ∥_{2}^{2}$ , the supremum of the above quantity under the constraint $∥ W ∥_{F} \leq R$ must be obtained when $∥ w_{t_{0}} ∥_{2} = B_{1}$ for some $t_{0} \in [H]$ , and $∥ w_{t} ∥_{2} = 0$ for all $t \neq t_{0}$ . Hence

	$sup ∥ W^{(0)} ∥_{F} \leq B_{1} ∥ ∥ m \sum i = 1 σ_{i k} {^h}_{i} (W^{(0)}) {∥ ∥}_{2}$
	$= sup ∥ w ∥_{2} = B_{1} (m \sum i = 1 σ_{i k} \sum v \in N (i) {^A}_{i v} h_{v} (w)) .$

Let $n_{s} (j)$ be the $s -$ th neighbor number of node $j$ ( $s \in [q]$ , $j \in [m]$ ). Recall $q := max | N (j) |, j \in [m]$ , $M_{s} = {max}_{i \in [m]} | {^A}_{i v} |$ with $v \in N (i)$ , therefore

	$E_{σ} [K \sum k = 1 sup ∥ W^{(0)} ∥_{F} \leq B_{1} ∥ ∥ m \sum i = 1 σ_{i k} {^h}_{i} (W^{(0)}) {∥ ∥}_{2}]$
	$= E_{σ} [K \sum k = 1 sup ∥ w ∥_{2} = B_{1} (m \sum i = 1 σ_{i k} \sum v \in N (i) {^A}_{i v} h_{v} (w))]$
	$\leq E_{σ} [K \sum k = 1 sup ∥ w ∥_{2} = B_{1} (q \sum s = 1 M_{s} m \sum i = 1 σ_{i k} h_{n_{s} (i)} (w))] .$

Applying the conclusion $sup (\sum_{s} a_{s}) \leq \sum_{s} sup (a_{s})$ and contraction property of Rademacher complexity again, we have

	$E_{σ} [K \sum k = 1 sup ∥ w ∥_{2} = B_{1} (q \sum s = 1 M_{s} m \sum i = 1 σ_{i k} h_{n_{s} (i)} (w))]$

	$\leq q \sum s = 1 M_{s} E_{σ} [K \sum k = 1 sup ∥ w ∥_{2} = B_{1} m \sum i = 1 σ_{i k} (\sum j \in N (n_{s} (i)) {^A}_{n_{s} (i) j} ⟨ x_{j}, w ⟩)]$
	$= q \sum s = 1 M_{s} E_{σ} [K \sum k = 1 sup ∥ w ∥_{2} = B_{1} ⟨ m \sum i = 1 σ_{i k} \sum j \in N (n_{s} (i)) {^A}_{n_{s} (i) j} x_{j}, w ⟩]$
	$\leq B_{1} q \sum s = 1 M_{s} E_{σ} [K \sum k = 1 ∥ ∥ m \sum i = 1 σ_{i k} \sum j \in N (n_{s} (i)) {^A}_{n_{s} (i) j} x_{j} {∥ ∥}_{2}] .$

Therefore, we only need to estimate the term

E_{σ} [K \sum k = 1 ∥ ∥ m \sum i = 1 σ_{i k} \sum j \in N (n_{s} (i)) {^A}_{n_{s} (i) j} x_{j} {∥ ∥}_{2}] .

Applying Cauchy-Schwartz inequality yields that

	$E_{σ} [K \sum k = 1 ∥ ∥ m \sum i = 1 σ_{i k} \sum j \in N (n_{s} (i)) {^A}_{n_{s} (i) j} x_{j} {∥ ∥}_{2}]$
	$\leq \sqrt{E_{σ} (K \sum k = 1 ∥ ∥ m \sum i = 1 σ_{i k} \sum j \in N (n_{s} (i)) {^A}_{n_{s} (i) j} x_{j} {∥ ∥}_{2})^{2}}$
	$\leq \sqrt{E_{σ} K K \sum k = 1 ∥ ∥ m \sum i = 1 σ_{i k} \sum j \in N (n_{s} (i)) {^A}_{n_{s} (i) j} x_{j} {∥ ∥}_{2}^{2}}$
	$\leq K \sqrt{m \sum i = 1 ∥ ∥ \sum j \in N (n_{s} (i)) {^A}_{n_{s} (i) j} x_{j} {∥ ∥}_{2}^{2}},$

where the last inequality is due to the i.i.d condition of Rademacher sequences. Plugging the conclusion of Lemma 3 into the above term leads to

	$E_{σ} [K \sum k = 1 ∥ ∥ m \sum i = 1 σ_{i k} \sum j \in N (n_{s} (i)) {^A}_{n_{s} (i) j} x_{j} {∥ ∥}_{2}]$
	$\leq K R \sqrt{q m},$

and

^R (F_{B_{1}, B_{2}}) \leq \frac{\sqrt{2 q} K B_{1} B_{2} R \sum_{s = 1}^{q} M_{s}}{\sqrt{m}} .

This completes the proof by combining with Lemma 2.

References

[1] J. Bruna, W. Zaremba, A. Szlam, and Y. Lecun, “Spectral networks and locally connected networks on graphs,” in ICLR, 2014.
[2] M. Gori, G. Monfardini, and F. Scarselli, “A new model for learning in graph domains,” in IJCNN, 2005.
[3] W. L. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in NIPS, 2017.
[4] J. Sun, W. Guo, D. Zhang, Y. Zhang, F. Regol, Y. Hu, H. Guo, R. Tang, H. Yuan, X. He, and M. Coates, “A framework for recommending accurate and diverse items using bayesian graph convolutional neural networks,” Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
[5] S. Casas, C. Gulino, R. Liao, and R. Urtasun, “Spagnn: Spatially-aware graph neural networks for relational behavior forecasting from sensor data,” 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 9491–9497, 2020.
[6] J. M. Stokes, K. Yang, K. Swanson, W. Jin, and J. J. Collins, “A deep learning approach to antibiotic discovery,” Cell, vol. 180, no. 4, pp. 688–702.e13, 2020.
[7] L. Yao, C. Mao, and Y. Luo, “Graph convolutional networks for text classification,” in AAAI, 2019.
[8] T. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” ICLR, 2017.
[9] J. Zhu, “Max-margin nonparametric latent feature models for link prediction,” in ICML, 2012.
[10] S. Ivanov and L. Prokhorenkova, “Boost then convolve: Gradient boosting meets graph neural networks,” in ICLR, 2021.
[11]
Q. Li, Z. Han, and X.-M. Wu, “Deeper insights into graph convolutional networks for semi-supervised learning,” in
AAAI, 2018.
[12] K. Sun, Z. Lin, and Z. Zhu, “Adagcn: Adaboosting graph convolutional networks into deep models,” in ICLR, 2021.
[13] S. Dz̆eroski and B. Z̆enko, “Is combining classifiers with stacking better than selecting the best one?” Machine Learning, vol. 54, pp. 255–273, 2004.
[14] M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst, “Geometric deep learning: Going beyond euclidean data,” IEEE Signal Processing Magazine, vol. 34, no. 4, pp. 18–42, 2017.
[15] B. Xu, H. Shen, Q. Cao, Y. Qiu, and X. Cheng, “Graph wavelet neural network,” in ICLR, 2019.
[16] M. Li, Z. Ma, Y. G. Wang, and X. Zhuang, “Fast haar transforms for graph neural networks,” Neural Networks, vol. 128, pp. 188–198, 2020.
[17] A. Sandryhaila and J. Moura, “Discrete signal processing on graphs,” IEEE Transactions on Signal Processing, vol. 61, no. 7, pp. 1644–1656, 2013.
[18]
D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst, “The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains,”
IEEE Signal Processing Magazine, vol. 30, no. 3, pp. 83–98, 2013.
[19] D. K. Hammond, P. Vandergheynst, and R. Gribonval, “Wavelets on graphs via spectral graph theory,” Applied and Computational Harmonic Analysis, vol. 30, no. 2, pp. 129–150, 2011.
[20] M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional neural networks on graphs with fast localized spectral filtering,” in NIPS, 2016.
[21] F. Wu, T. Zhang, A. Souza, C. Fifty, T. Yu, and K. Q. Weinberger, “Simplifying graph convolutional networks,” in ICML, 2019.
[22]
Q. Li, Z. Han, and X. M. Wu, “Deeper insights into graph convolutional networks for semi-supervised learning,” in
AAAI, 2018.
[23] G. Li, M. Mueller, A. Thabet, and B. Ghanem, “Deepgcns: Can gcns go as deep as cnns?” in ICCV, 2019.
[24] S. Luan, M. Zhao, X. W. Chang, and D. Precup, “Break the ceiling: Stronger multi-scale deep graph convolutional networks,” in NIPS, 2019.
[25] D. Wolpert, “Stacked generalization,” Neural Networks, vol. 5, pp. 241–259, 1992.
[26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, u. Kaiser, and I. Polosukhin, “Attention is all you need,” in NIPS, 2017, pp. 6000–6010.
[27] V. Mnih, N. Heess, A. Graves, and K. Kavukcuoglu, “Recurrent models of visual attention,” in NIPS, 2014.
[28]
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in
ICLR, 2015.
[29] W. Yin, H. Schütze, B. Xiang, and B. Zhou, “Abcnn: Attention-based convolutional neural network for modeling sentence pairs,” Transactions of the Association for Computational Linguistics, vol. 4, pp. 259–272, 2016.
[30] R. Schapire, Y. Freund, P. Barlett, and W. S. Lee, “Boosting the margin: A new explanation for the effectiveness of voting methods,” in ICML, 1997.
[31] S. Lv, “Generalization bounds for graph convolutional neural networks via rademacher complexity,” 2021.
[32] P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Gallagher, and T. Eliassi-Rad, “Collective classification in network data,” AI Mag., vol. 29, pp. 93–106, 2008.
[33] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Lio’, and Y. Bengio, “Graph attention networks,” in ICLR, 2018.
[34] J. Klicpera, A. Bojchevski, and S. Günnemann, “Predict then propagate: Graph neural networks meet personalized pagerank,” in ICLR, 2019.
[35] K. Pal and B. Patel, “Data classification with k-fold cross validation and holdout accuracy estimation methods with 5 different machine learning techniques,” 2020 Fourth International Conference on Computing Methodologies and Communication (ICCMC), pp. 83–87, 2020.
[36] J. Friedman, “Greedy function approximation: A gradient boosting machine.” Annals of Statistics, vol. 29, pp. 1189–1232, 2001.
[37] Y. Freund and R. E. Schapire, “A short introduction to boosting,” Journal of Japanese Society for Artificial Intelligence, vol. 14, pp. 771–780, 1999.
[38] F. Schwenker, “Ensemble methods: Foundations and algorithms,” pp. 77–79, 2013.
[39] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in ICLR, 2015.
[40] L. V. D. Maaten and G. E. Hinton, “Visualizing data using t-sne,” Journal of Machine Learning Research, vol. 9, pp. 2579–2605, 2008.
[41] A. Maurer, “A vector-contraction inequality for rademacher complexities,” in International Conference on Algorithmic Learning Theory, 2016.
[42] P. L. Bartlett and S. Mendelson, “Rademacher and gaussian complexities: Risk bounds and structural results,” in Conference on Computational Learning Theory & and European Conference on Computational Learning Theory
, 2001.
[43]
B. Gao and L. Pavel, “On the properties of the softmax function with application in game theory and reinforcement learning,” 2017.

SStaGCN: Simplified stacking based graph convolutional networks

Authors

Multi-scale Graph Convolutional Networks with Self-Attention

Border-SegGCN: Improving Semantic Segmentation by Refining the Border Outline using Graph Convolutional Network

Graph Highway Networks

Graph-Driven Generative Models for Heterogeneous Multi-Task Learning

Turn Tree into Graph: Automatic Code Review via Simplified AST Driven Graph Convolutional Network

Multivariate Relations Aggregation Learning in Social Networks

Poisson Kernel Avoiding Self-Smoothing in Graph Convolutional Networks

I Introduction

Ii Related Work

Iii The proposed approach: SStaGCN

Iii-a Graph convolutional networks

Iii-B Stacking

Iii-C The proposed approach

Iii-D Generalization Bound Analysis

Theorem 1

Remark 1

Iv Experiments

Iv-a Datasets

Iv-B Baselines

Iv-C Setting

Iv-D Results

Iv-E Visualization

V Conclusion

Vi Appendix

Lemma 1

Lemma 2

Lemma 3

Proof 1

Proof 2

References

SStaGCN: Simplified stacking based graph convolutional networks

Authors

Related Research

Multi-scale Graph Convolutional Networks with Self-Attention

Border-SegGCN: Improving Semantic Segmentation by Refining the Border Outline using Graph Convolutional Network

Graph Highway Networks

Graph-Driven Generative Models for Heterogeneous Multi-Task Learning

Turn Tree into Graph: Automatic Code Review via Simplified AST Driven Graph Convolutional Network

Multivariate Relations Aggregation Learning in Social Networks

Poisson Kernel Avoiding Self-Smoothing in Graph Convolutional Networks

This week in AI

I Introduction

Ii Related Work

Iii The proposed approach: SStaGCN

Iii-a Graph convolutional networks

Iii-B Stacking

Iii-C The proposed approach

Iii-D Generalization Bound Analysis

Theorem 1

Remark 1

Iv Experiments

Iv-a Datasets

Iv-B Baselines

Iv-C Setting

Iv-D Results

Iv-E Visualization

V Conclusion

Vi Appendix

Lemma 1

Lemma 2

Lemma 3

Proof 1

Proof 2

References