HopGAT: Hop-aware Supervision Graph Attention Networks for Sparsely Labeled Graphs

04/09/2020 ∙ by Chaojie Ji, et al. ∙ 0

Due to the cost of labeling nodes, classifying a node in a sparsely labeled graph while maintaining the prediction accuracy deserves attention. The key point is how the algorithm learns sufficient information from more neighbors with different hop distances. This study first proposes a hop-aware attention supervision mechanism for the node classification task. A simulated annealing learning strategy is then adopted to balance two learning tasks, node classification and the hop-aware attention coefficients, along the training timeline. Compared with state-of-the-art models, the experimental results proved the superior effectiveness of the proposed Hop-aware Supervision Graph Attention Networks (HopGAT) model. Especially, for the protein-protein interaction network, in a 40 from 98.5 also demonstrate the effectiveness of supervised attention coefficient and learning strategies.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Node classification is used to predict the class of unlabeled nodes given a partially labeled graph. Node classification is one of the most important applications in analyzing graphs in various areas, including document classification in social science Atwood and Towsley (2016), disease prediction in bioinformatics Yue et al. (2019), and department classification of an employer in communication networks Zhang et al. (2019c, b). However, labeling nodes for a training task is time consuming and sometimes very expensive Li et al. (2019b). For example, to initially acquire the disease label of a group of disease-causing genes, one must sequence sufficient patient and normal samples Nguyen and Ho (2012). Being able to predict a node class in a sparsely labeled graph while maintaining the prediction accuracy deserves more attention Dornaika and Weng (2019).

To address insufficiently labeled data, researchers widely adopt semi-supervised learning, in which both labeled and unlabeled neighbor nodes are utilized

Yang et al. (2016); Kim et al. (2019). A general assumption in homogeneous graph network research is that a node usually possesses more similar information to its immediate neighbors Zhang et al. (2019a). These studies learn node representations on the surrounding nodes or link information via the convolution operation Defferrard et al. (2016); Li et al. (2019a). Furthermore, graph attention networks (GATs) Veličković et al. (2018) and various variants Fan et al. (2020); Liu et al. (2019) have been proposed to quantify the closeness of node pairs through attention Lee et al. (2019).

Figure 1: The classification accuracy highly depends on the four colored labeled nodes.

However, the deep learning algorithms

Zhang et al. (2018); Wu et al. (2018), such as Graph Convolutional Networks (GCNs) Kipf and Welling (2017); Bruna et al. (2014) and GATs Veličković et al. (2018); Xiong et al. (2019), strongly depend on the labeled nodes to train a prediction model, and thus, the performance is limited by the scale of the labeled data. As seen in Figure 1, colored nodes are labeled, while gray nodes are unlabeled. To predict the labels of Node C, the algorithms must fully take advantage of the information from Nodes A, B, D and E. Therefore, the prediction accuracy strongly depends on the four labeled color nodes.

The number of hops, namely, the hop value, is adopted to describe the distance or neighborhood relationship between two nodes in the graph network. The general assumption is that neighbors with different hop values have different influences on their center node. For classification tasks on sparsely labeled graphs, in which a center node has very few labeled neighbors, it is essential to learn more information from more neighbors with different hop values. The key point is how the algorithm learns sufficient semantic information from neighbors with different hop values.

1.1 Motivation

In this study, we make two key observations.

Observation 1: The class labels of neighboring nodes with a smaller hop value are more likely to be consistent with those of their center nodes in the homogeneous graph network.

Figure 2: Relation between hop value and class label.

We examined the data set Cora Ioffe and Szegedy (2015), which is usually used in scientific publication classification tasks, and recorded the label consistence rate, the proportion of neighboring nodes having the same class label as their center nodes. We recorded this consistency rate under different sets of neighboring nodes with a given hop value. In Figure 2, the x-axis is the given hop value, and the y-axis is the label consistency rate.

From the visualization, we can observe that the neighboring nodes with a smaller hop value are more consistent with their center nodes. This is consistent with the general assumption in homogeneous graph networks that a node is more likely similar to its immediate neighbors. This experiment further extends the assumption to neighbors with larger hop values and proves the rationality of introducing hop values to discriminate the closeness of two nodes.

Observation 2: General attention models are unable to automatically learn sufficient semantic information from neighbors with different hop values.

Based on the motivation from Observation 1, we checked the attention coefficients, which should be different between nodes with different hop values since the neighboring nodes with a smaller hop value are more consistent with their center nodes.

Figure 3: Attention coefficient distribution produced from a trained GAT.

We trained a GAT on the Cora citation dataset with the hyper-parameter settings mentioned in Veličković et al. (2018)

. The max hop value is set to 2. All attention coefficients produced by the GATs during the different training epochs are visualized in Figure

3

. In this figure, the horizontal axis shows the value of the attention coefficients, the vertical axis records the occurrence number of the coefficients, and the z-axis represents the training epochs. This figure illustrates a Gaussian distribution with only one peak. However, we expect a distribution that has multiple peaks or clear boundaries corresponding to different hop values. This implies that unsupervised attention mechanisms are generally unable to automatically learn the correlation or semantic information for nodes with different hop values in graph networks. This observation is consistent with the work

Knyazev et al. (2019) in that attention should be supervised under different conditions.

1.2 Contributions

In this study, we suggest a HopGAT model to address node classification on a sparsely labeled graph. Compared with the general node classification error, we jointly supervise hop-aware attention coefficients in the loss function. Our contributions are described in the following:

  • To the best of our knowledge, this paper is the first study to propose a hop-aware attention supervision for the node classification task.

  • We encoded hop values and embedded them into the graph nodes. Subsequently, the graph nodes were simultaneously encoded with semantics and graph structure information.

  • We proposed a simulated annealing strategy to simultaneously balance two learning tasks, node classification and attention coefficients, along the training timeline.

  • Experimental results prove the effectiveness of the proposed HopGAT model over the state-of-the-art baselines and quantify our improvement on the sparsely labeled datasets. In addition, the extensive experiments also show the effectiveness of supervised attention coefficients and learning strategy.

2 Related Work

2.1 Weakly-supervised Learning in Graph

Compared with traditional supervised learning, weakly supervised learning aims to address the situation in which precise or sufficient labels are unachievable Carbonneau et al. (2018); Frenay and Verleysen (2014). In the graph domain, semi-supervised learning is widely used to address incomplete labels in a graph, as in Shi et al. (2019); Xu et al. (2019). These studies mainly focus on graph representation. Vashishth et al. proposed ConfGCN to introduce the concept of the locality of labels Vashishth et al. (2019)

. They used additional label distribution and co-variance matrices derived from limited labeled nodes. None of these studies took advantage of implicit information in unlabeled neighbors and their hop values. Yang et al. proposed a method whereby node embedding is trained to predict the class label

Yang et al. (2016). They used neighbor context to train the node embedding similar to a Skipgram-like model. The coefficients between two nodes are not computed explicitly or trained in the prediction model. Inspired by the locality of labels in ConfGCN and neighbor context in Yang’s methods, we conducted a further analysis of the changing of the node labels with different hop values. Based on the observation that neighboring nodes with smaller hop values are more likely to be consistent with their center nodes, we embed the hop value information in the classification function in this study.

2.2 Attention in Graph

Given the trend whereby convolution operations are being generalized into arbitrary graph networks, more effective methods to aggregate neighboring nodes and locate “closest” nodes from center nodes are desired, as in Lee et al. (2018); Wang et al. (2018). An attention mechanism aids a model to “focus on the most relevant parts of the input to make decisions” Shaw et al. (2018); Vaswani et al. (2017). Although attention is widely used in locating the closest nodes and have achieved state-of-the-art performance Shanthamallu et al. (2019), the studies in Choi et al. (2017) showed that attention should be supervised to obtain a better performance under different conditions. In this study, we jointly supervise the hop-aware attention coefficients and the node classification error in the loss function to better train an algorithm given insufficiently labeled nodes.

2.3 Supervision on Attention in Graph

To better understand attention mechanism in graph convolution network, Knyazev et al. proposed ChebyGIN and imposed a supervision on attention coefficients as a controlled environment Knyazev et al. (2019). They noticed the influence of attention and mentioned that the accuracy of a model could depend exponentially on attention correctness. The proposed mechanism of attention supervision is goal-directed which can’t be directly applied in other tasks, such as node classification.

3 Preliminary

Before introducing our proposed method, we provide a brief overview of the semi-supervised GATs which are composed of several single graph attentional layer. We only describe a single graph attention layer here.

Given a graph, the input to current layer are defined as node features, i.e. , where is the number of nodes.

The attention coefficients between two nodes can be calculated as follows:

(1)

where is the exponential function, is the neighboring nodes of in the graph, and

is a function used to estimate the importance of one node to another.

Once obtained, the attention coefficients of node are used to compute a linear combination of the features corresponding to its neighboring nodes, which can be considered as the updated output features for node :

(2)

where is a weight matrix.

Similar to the Transformer Vaswani et al. (2017), multi-head attention is employed in GATs. independent attention mechanisms execute a transformation as in Equation 2. Then, the produced features are concatenated as

(3)

where denotes the concatenation operation, and corresponds to the th head.

For the final layer, GATs employ averaging and nonlinearity to a specific task:

(4)

4 Method

Figure 4: Overall framework of an Hop-GAT.

In this section, we describe the architecture of our model - HopGAT. The model has three main components: a hop encoding and attention mechanism, attention supervision, and a learning strategy, as shown in Figure 4. The other modules shown in the figure are inherited from the GATs.

In component 1, we first encode the hop value into a vector and embed the vector into each node feature. Then, we calculate the attention coefficients of each center node to its neighboring nodes with hop information. Finally, we apply multi-head hop attention for each layer, which can draw different features of the graph network. In the next layer, the node features are updated according to features from the previous layer, and the attention coefficients for each node pair are delivered to the attention supervision. In component 2, all the attention coefficients from every head are collected. The gap between the computed coefficients and the defined ground-truth coefficients are summed to form an attention loss, which will be one part of the total loss function used to train the prediction model. In the last layer, the node classification loss is calculated. Once the attention loss and node classification loss are obtained, our learning strategy is performed in component 3 to balance these two types of losses during the training procedure.

4.1 Attention Mechanism

4.1.1 Hop Encoding

Existing attention-based graph methods usually specify the maximum hop value between a center node and its neighbors, e.g. 1, in GATs. Then, the numerical hop value in a graph is coarsely equivalent to a Boolean variable representing whether a node is a neighbor of another node. However, as we have noted in Observation 1, the class labels of neighboring nodes with a smaller hop value are more likely to be consistent with those of their center nodes in a homogeneous graph network, and the hop value offers more information than a Boolean variable.

In this study, a hop value is used to express the closeness or similarity of a node to its center node. Therefore, we encode a hop value into a -dimensional vector as :

(5)
(6)

where is the pre-defined maximal hop value, is a maximum value function, is the dimension of the hop embedding, is the index of the dimensions, is the corresponding hop value, and are sine and cosine functions, respectively, and represents the hop encoding for the hop value .

Through this definition, not only the absolute hop value but also the relative hop value can be learned since, as a fixed offset , can be transformed as a linear function of . Since the number of heads in each layer can be different, the dimensions of nodes in each hidden layer are also different. Therefore, the dimension of the hop embedding in each layer may be different.

4.1.2 Attention Mechanism

We propose two attention mechanisms based on the above hop encoding: product-based attention and addition-based attention.

For a node pair , and are the center and neighboring nodes, respectively. is the hop value from to . The hop encoding for the th layer is written as , where each line represents a hop encoding for a hop value.

  • Product-based Attention This attention mechanism is mainly based on the dot product between the embeddings of two nodes.

    (7)

    indicates the importance of node to node , , and

    are three independent single-layer feedforward neural networks,

    denotes the dot product operation, and is a function that looks up the hop encoding for the hop value in .

  • Addition-based Attention This mechanism is based on the addition operation on two node embeddings. The attention coefficients are formulated as follows:

    (8)

    where the symbol

    is a concatenation operation. The LeakyReLU activation function is used to execute the non-linearization operation.

To make coefficients easily comparable, we normalize them across all neighbors using the softmax function:

(9)

where is the neighboring node set of node . Once is obtained, we use Equations 3 - 4 to perform the final node classification.

The hop value of self-connection is defined as 0, and it is also encoded into a non-zero hop encoding vector. Through this design, all attention coefficients are uniformly calculated according to the node features and hops.

4.2 Attention Supervision

4.2.1 Ground-truth Attention

The hop value represented in the graph data is a label-free and effective indicator for quantifying the correlation between two nodes. Specifically, the correlation between two directly connected nodes in the graphs should be assigned a larger value, while the coefficient between two indirectly connected or disconnected nodes is expected to have a smaller value. In addition, a prior investigation showed that “classification accuracy depends exponentially on attention correctness” Knyazev et al. (2019) and that the general attention models cannot effectively automatically learn sufficient semantic information from neighbors with different hop values. All of this motivates us to develop a hop-aware approach to supervise the training process for attention coefficients between two nodes.

Generally, the correlation between two nodes becomes weaker when their hop value is over a certain value. Therefore, we set a boundary for the hop value. The coefficients between the center node and its neighboring nodes are uniformly grouped into a default value when the hop value reaches the maximum value, , which means a weak connection between two nodes. We formulate the definition of ground-truth attention as follows:

(10)

This can be interpreted as follows: the larger the hop value is, the smaller the ground-truth attention.

When the hop value is greater than two, a negative ground-truth attention value will be assigned. This negative value does not indicate a negative association between these two nodes. This coefficient has not been normalized by the softmax function as in Equation 9. After normalization, the attention coefficients will be between 0 and 1, indicating the strong and weak correlation between two nodes.

To be more accurate, we use to denote the attention coefficient between nodes and of the th head in the th layer, which are produced during the training procedure as in Equations 7 and 8. We use the mean square error to control the distance between the ground-truth attention and the computed coefficients .

(11)

will be used as a part of the loss function to supervise the training process for the attention coefficients between two nodes.

4.2.2 Sample Strategy

In total, there are attention coefficients in the calculation of . To decrease the computation cost, especially in a graph with a large number of nodes, we thus proposed a random sampling strategy, , with which a subset of node pairs is sampled with sample ratio .

The number of node pairs with is greater than that with less than . For example, in the Citeseer dataset, there are approximately 12,000 node pairs with less than 2 hops, whereas there is approximately 11,000,000 pairs with more hops. For balancing the distribution between node pairs with hop values of greater or less than , only samples from the node pairs with hop values greater than . Furthermore, we sample each batch differently to guarantee the diversity of the training data. is calculated as follows:

(12)

4.3 Learning Strategy

In addition to the loss function used to supervise the attention coefficients, we also include the general node classification loss to measure the classification error.

The final objective for optimization is the linear combination of these two terms.

(13)

where is used to find a balance between node classification and attention supervision losses.

Inspired by the analysis that attention coefficients are more likely to be imprecise at the beginning of training Knyazev et al. (2019), more powerful supervision of the attention should be imposed at the early stage of the training. From another perspective, is a “strong” label and is closely related to the final task goal, i.e., node classification, while is auxiliary. Thus, a consideration of balancing them with time is necessary. A simulated annealing procedure is adopted to help the model find the best combination of these two parts of the total loss function.

We first define the transformation of the temperature along the training time:

(14)

where and are the initial and final temperatures, respectively; indicates the temperature at the th time step; is the decay rate; and , and

are all pre-defined hyperparameters.

Then, and are biased through along the training time as follows:

(15)

where is the minimum function. is a hyperparameter designed to prevent a sharp increase of at the tail of the training process. The learning procedure will be explained in greater detail in the experiment section.

measures the classification error and is applied to all the labeled nodes during the training procedure. If the labeled nodes in the graph network are insufficient, the trained model will not be sufficiently accurate. In this study, the hop value is introduced as an effective supplementary to the insufficient labels to train the model but without increasing the labeling cost.

5 Experiments

In this section, we evaluate the performance of the proposed hop-aware attention supervision model to address the node classification task on a sparsely labeled graph network. We also investigate the effectiveness of the supervised attention coefficients and our learning strategy.

5.1 Dataset

This experiment includes two types of tasks: inductive learning and transductive learning. For the semi-supervised tasks, if the unlabeled test nodes do not participate in the training procedure, we call this task inductive learning, whereas if the unlabeled test data are all observed and utilized during the training phase, the task is transductive learning.

Cora, Citeseer and PubMed are chosen as our benchmark datasets for transductive tasks Ioffe and Szegedy (2015). In all of these datasets, nodes denote documents, and edges correspond to citation relations. Node features are elements of a bag-of-words representation of a document. The task is to predict the unique document class among multiple documents. The protein-protein interaction (PPI) dataset is used to evaluate inductive tasks as in Sen et al. (2008). Each node is a protein. Positional and motif gene sets and immunological signatures are used to represent a protein. It is a multi-label task that simultaneously predicts multiple protein functions (labels). We used the preprocessed data from Hamilton et al. (2017) in our experiments.

We evaluate how the proposed algorithm works on the insufficiently labeled graph network. Therefore, we change the proportions of the labeled nodes in the training set. We then reorganize all the datasets.

#Nodes #G #F # C #VN #TN # Training Nodes with Label Rate (%)
20 40 60 80 100
Cora 2708 1 1433 7 500 1000 242 484 725 967 1208
Citeseer 3327 1 3703 6 500 1000 363 725 1008 1450 1812
PubMed 19717 1 500 3 500 1000 3644 7287 10931 14574 18217
PPI 56944 24 50 121 6514 5524 8982 17963 26944 35925 44906
Table 1: Number of randomly sampled nodes. Column #G, #F, #C, #VN and #TN denote the number of graphs, features, classes, validation nodes and test nodes, respectively.

The validation and test sets completely follow the experimental setup of Veličković et al. (2018). In all transductive tasks, 500/1000 nodes serve for the validation/test sets; 6514/5524 nodes (in 2/2 graphs) are used for the validation/test sets in the inductive tasks. The remaining nodes are placed into the training set. Then, we randomly sample nodes at rates of 20%, 40%, 60%, 80%, and 100% without replacement from the training set for each dataset. We reserve the labels of the sampled nodes but mask the labels of other nodes. Subsequently, we obtained 5 variants for each dataset. The details of the number of nodes, classes, graphs and features in each dataset are listed in Table 1.

5.2 Experimental Setup

We compared the proposed HopGAT against state-of-the-art methods. GATs 111https://github.com/PetarV-/GAT, GCNs 222https://github.com/tkipf/gcn Kipf and Welling (2017) and ConfGCN 333https://github.com/malllabiisc/ConfGCN Vashishth et al. (2019) were chosen for the transductive tasks. GraphSAGEs444https://github.com/williamleif/GraphSAGE Hamilton et al. (2017)

and GATs were chosen for the inductive tasks. Furthermore, we selected different variants of GraphSAGEs, i.e., GraphSAGE with the mean-based aggregator (Mean), LSTM-based aggregator (Seq), max-pooling aggregator (Maxpool), mean-pooling aggregator (Meanpool) and GCN-based aggregator (GCNagg). For each task, we run the same experiment five times and record the average performances and the standard deviations. For the inductive tasks, we used the metric of the micro-averaged

score and the accuracy for the transductive tasks instead.

Attention # L # Heads # Features BS LR
Cora 0.2 0.0 0.2 0.0001 Addition 2 [8,1] [8,7] 1 0.005
Citeseer 0.6 0.2 0.6 0.0 Addition 2 [8,1] [8,6] 1 0.005
PubMed 0.0 0.0 0.0 0.0 Addition 2 [8,8] [8,3] 1 0.01
PPI 0.0 0.0 0.0 0.0 Product 3 [4,4,6] [256,256,121] 2 0.005
Table 2: Common Hyperparameters of GATs and HopGAT on different tasks. Column #L, #BS and #LR denote the number of layers, batch size and learning rate, respectively.
GCN ConfGCN
Cora Citeseer PubMed Cora Citeseer Pubmed
dp 0.1 0.4 0.0 0.8 0.3 0.0
1e-4 0.0 0.0 0.01 0.05 0.0
Table 3: Hyperparameters of GCNs and ConfGCNs in transductive tasks.

We applied dropout Srivastava et al. (2014), skip connections He et al. (2016) and regularization techniques to alleviate over-fitting. For each layer in the HopGAT model, we applied three dropout units: when receiving the updated node representation from the previous layer, archiving the normalized attention coefficients in Equation 2, and obtaining the transformed node representation in Equation 2. We denote these dropout rates as , and , respectively. The skip connections were employed across the intermediate layers when the number of layers was greater than two. An exponential linear unit (ELU) Clevert et al. (2016) instead of a function in Equation 2 is implemented when the computation does not occur in the last layer. We applied a single-layer feedforward neural network when the dimension of the inputs is not equal to the number of features in the last layer. We fixed the number of output units in the single-layer feedforward neural networks to 1, i.e., , and in Equation 7. The common parameters of GATs and HopGATs are presented in detail in Table 2. We mainly adjusted the dropout rate and regularization for GCN and ConfGCN, which are listed in Table 3. The coefficient of in ConfGCN’s objective function was set to 1.0 in the PubMed dataset. The other hyperparameters not mentioned here were all derived from their original publications, i.e., GCN Kipf and Welling (2017), ConfGCN Vashishth et al. (2019) and GraphSAGEs Hamilton et al. (2017).

We used the Adam SGD optimizer Kingma and Ba (2015). An early stopping strategy is applied to the loss for the node classification and also the accuracy (or micro-F1) on the validation nodes with a patience of 100 epochs in all experiments. For the learning strategy, the initial temperature is set to 100, and the lowest temperature is 1. The decay rate for the Cora dataset is 0.95, and it is 0.85 for the other datasets. The saturation gamma is 0.25. For the sampling strategy, we used different percentages to sample node pairs - 0.0003, 0.0005 and 0.0001 for Cora, Citeseer and PubMed, respectively.

5.3 Results

5.3.1 Inductive Task

GCNagg Mean Meanpool Maxpool Seq GAT Hop-GAT Imp
PPI(100%) 51.60.5 58.00.9 59.00.5 60.20.7 61.20.5 97.30.2 98.50.1 +1.2
PPI(80%) 45.30.7 55.60.2 52.81.0 52.60.7 56.20.6 96.80.2 97.80.3 +1.0
PPI(60%) 46.90.4 54.40.7 52.51.4 51.02.3 54.00.5 95.20.4 96.70.6 +1.5
PPI(40%) 48.80.8 52.90.2 49.80.4 48.80.2 52.20.9 91.50.4 94.61.8 +3.1
PPI(20%) 44.10.3 50.50.4 43.21.1 43.61.5 48.40.7 83.30.3 88.61.2 +5.3
Table 4: F1 score of inductive tasks. Symbols Imp denotes the improvement.

The results for the inductive tasks are listed in Table 4. The improvement column records the improved values of HopGAT against GATs, which shows better performance compared to other baseline methods.

Compared with GATs, the results demonstrate that HopGATs obtain a significant gain across all datasets with different proportions of labeled nodes. Specifically, the smallest improvement is 1.0% for the 80% label rate, and the highest improvement is 5.3% for the 20% label rate. It is observed that given fewer labeled nodes, a higher performance gain is achieved, from 1.2% to 5.3% - except for a small fluctuation at the 80% label rate. The results show that the proposed model is effective in addressing the inductive task on the sparsely labeled graph network.

Another important observation from the table is that we cannot achieve a proportional performance gain by labeling additional nodes. Labeling more nodes, e.g., from 20% to 40%, the performance gains is nearly 6%. With a 40% labeled graph, the performance loss is only 3.9%, from 98.5% to 94.6%, compared to the fully labeled graph. More labeled nodes results in smaller performance gains. Therefore, if necessary, we should balance the cost of labeling more nodes and performance gain.

5.3.2 Transductive Tasks

GCN Conf-GCN GAT Hop-GAT Improvement
Cora (100%) 87.2 0.4 87.5 0.4 88.1 0.3 88.1 0.4 + 0.0
Cora (80%) 86.8 0.2 87.3 0.2 86.8 0.2 87.3 0.4 + 0.5
Cora (60%) 85.8 0.2 86.5 0.4 86.5 0.2 87.1 0.1 + 0.6
Cora (40%) 84.5 0.3 86.1 0.1 86.0 0.4 86.5 0.3 + 0.5
Cora (20%) 82.4 0.3 83.0 0.2 83.1 0.3 83.6 0.2 + 0.5
Citeseer (100%) 78.8 0.2 77.5 0.2 78.4 0.9 79.5 0.3 + 1.1
Citeseer (80%) 78.0 0.3 76.8 0.4 77.4 0.9 78.1 0.4 + 0.7
Citeseer (60%) 76.9 0.5 76.1 0.7 77.0 0.3 77.8 0.5 + 0.8
Citeseer (40%) 75.3 0.3 74.9 0.5 75.5 0.7 76.2 0.4 + 0.7
Citeseer (20%) 72.8 0.2 74.3 0.4 73.9 0.3 74.3 0.1 + 0.4
PubMed (100%) 87.3 0.1 85.9 0.3 87.5 0.7 88.9 0.2 + 2.4
PubMed (80%) 87.5 0.2 86.1 0.4 87.5 0.3 88.7 0.6 + 1.2
PubMed (60%) 86.9 0.2 86.0 0.4 86.9 0.2 88.3 0.2 + 1.4
PubMed (40%) 86.5 0.1 86.0 0.4 86.0 0.4 87.6 0.2 + 1.6
PubMed (20%) 86.5 0.2 85.1 0.3 86.2 0.2 87.3 0.4 + 1.1
Table 5: Accuracy of transductive tasks.

The results of all transductive tasks are shown in Table 5. Compared with GATs, the maximum improvement for Cora, Citeseer and PubMed are 0.6%, 1.1% and 2.4% respectively. The minimum improvements are 0.0%, 0.4% and 1.1% respectively. This proves the effectiveness of our model on transductive tasks.

We also investigate the improved average performance on individual datasets. The improved average accuracy is 0.42%, 0.74% and 1.54% on the Cora, Citeseer and PubMed datasets, respectively. These datasets include 2708, 3327, and 19717 nodes, respectively. This shows that the larger the graphs are, the more benefit that the proposed model can obtain.

We can observe that the greater the number of labeled nodes, the higher the performance gain obtained on Citeseer and PubMed. This is different from the inductive task, in which fewer labeled nodes results in higher performance gains. This could be caused by the large difference between the mechanisms of inductive and transductive tasks. In the inductive tasks, validation and test nodes are completely unseen in the entire training process, whereas they directly participate in the training process in the transductive tasks. This means that with the HopGAT model, the training nodes could learn more correlations from the validation/test nodes in the transductive tasks, even when the test nodes are not labeled. In addition, the smaller the dataset is, the stronger the randomness during the sampling, which can cause a fluctuation of the performance gains in certain cases.

5.3.3 Effectiveness of Supervised Attention Coefficients

(a) Attention distribution in the 1st layer.
(b) Attention distribution in the 2nd layer.
Figure 5: Attention distribution produced from the trained HopGAT.

We investigate the effectiveness of the supervised attention coefficients. We thus generate a visualization of the attention coefficients produced by the proposed HopGAT model. We trained on the training set of PubMed. We record the distribution of the produced attention coefficients during the different training epochs. Figure 5 (a) comes from a head of the 1st layer, and Figure 5 (b) is from the 2nd layer. Similar to Figure 3, the horizontal axis shows the value of the attention coefficients, and the vertical axis records the occurrence number of the coefficients. Compared to Figure 3, there are three clusters that correspond to the attention coefficients for 0, 1 and greater than 1 hop value from left to right.

Comparing Figure 5 (a) to Figure 5 (b), we noticed that the attention coefficients inside the 2nd layer have more clear boundaries among different hop values than those in the 1st layer. This is consistent with the expectation that the signal received from the deeper layer is much stronger than the first layers due to back propagation.

5.3.4 Effectiveness of Learning Strategy

In this section, we evaluate the proposed learning strategy. We train the HopGAT model on PPI with a 100% label rate and the hyperparameters mentioned above. Then, we record the entire training process as a learning curve of the loss for the node classification, attention coefficients, and micro-averaged score. We also visualize the changes of with increasing training epochs.

(a) Variation of .
(b) Learning curve between epoch and node classification loss.
(c) Denoised learning curve between epoch and attention coefficient loss.
(d) Learning curve between epoch and micro-F1 score.
Figure 6: The curves related to our proposed learning strategy.

The value of is used to balance the node classification and attention supervision loss functions. As shown in Figure 6 (a), at the early stage of training, i.e., the first 100 steps, more powerful supervision on the attention was imposed due to a larger . Once is relatively stable and when reasonable attention coefficients are established, the major energy of the learning strategy turns into alternatively, as shown in Figure 6 (b) and 6 (c). This is defined as mid-term - from 100 to 1,700 steps.

At the tail of the training process, from epoch 1,700 to 2600, we observed that the learning curve was reasonably smooth and that was stable and small. is a relatively large value, above 0.8 (one subtracts the value in Figure 6 (a)), and the node classification loss thus draws greater attention from the optimizer. The optimizer adjusts the gradient almost based on the and therefore may neglect the subsequent impact on , which could result in a sharp increase in . As shown in Figure 6 (c), we captured these fluctuations four times. We therefore introduce the saturation parameter to resist the sudden increase in . In this way, the fluctuations are handled appropriately, and better performance is achieved. Importantly, this type of fluctuation only occurs at the end of the training process, and it is not a necessary appearance for each training procedure.

In Figure 5, another interesting point is that during the first 40 epochs, is relatively large, and the HopGAT thus pays more attention to the adjustment of and impacts the distribution of the attention coefficients. We can observe the emergence of the three separate clusters in this phase. After the 40 epochs, starts to be restricted, and the boundaries of the clusters become more clear. This provides evidence that once reasonable supervision of the attention coefficients in the early phase is applied, the subsequent learning of the node classification can be jointly performed.

6 Conclusion and Future Work

This paper proposes a hop-aware attention supervision mechanism for the node classification task. Different from the previous works, we consider the influence of the hop values between a center and its neighbor nodes. Furthermore we jointly supervise the hop-aware attention coefficients and node classification error in the loss function, by which the loss function could be trained from more information of the context nodes. This method achieves state-of-the-art classification performance. In particular, it seems more effective for the inductive task in a graph with very few labeled nodes.

In the future, there are two interesting works can be done: (1) exploring a general hop-aware model which not only performs on node classification but also on link prediction task. (2) exploring the application of hop value and attention supervision mechanism on the heterogeneous graph network.

References

  • J. Atwood and D. Towsley (2016)

    Diffusion-convolutional neural networks

    .
    In Proceedings of the Advances in Neural Information Processing Systems, pp. 1993–2001. Cited by: §1.
  • J. Bruna, W. Zaremba, A. Szlam, and Y. Lecun (2014) Spectral networks and locally connected networks on graphs. In Proceedings of the International Conference on Learning Representations, Cited by: §1.
  • M. Carbonneau, V. Cheplygina, E. Granger, and G. Gagnon (2018) Multiple instance learning: a survey of problem characteristics and applications. Pattern Recognition 77, pp. 329–353. Cited by: §2.1.
  • E. Choi, M. T. Bahadori, L. Song, W. F. Stewart, and J. Sun (2017) GRAM: graph-based attention model for healthcare representation learning. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 787–795. Cited by: §2.2.
  • D. Clevert, T. Unterthiner, and S. Hochreiter (2016) Fast and accurate deep network learning by exponential linear units (elus). In Proceedings of the International Conference on Learning Representations, Cited by: §5.2.
  • M. Defferrard, X. Bresson, and P. Vandergheynst (2016) Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of the Advances in Neural Information Processing Systems, pp. 3844–3852. Cited by: §1.
  • F. Dornaika and L. Weng (2019) Sparse graphs with smoothness constraints: application to dimensionality reduction and semi-supervised classification. Pattern Recognition 95, pp. 285–295. Cited by: §1.
  • X. Fan, M. Gong, Y. Xie, F. Jiang, and H. Li (2020) Structured self-attention architecture for graph-level representation learning. Pattern Recognition 100, pp. 107084. Cited by: §1.
  • B. Frenay and M. Verleysen (2014) Classification in the presence of label noise: a survey. IEEE Transactions on Neural Networks and Learning Systems (), pp. 845–869. External Links: Document, ISSN 2162-2388 Cited by: §2.1.
  • W. Hamilton, Z. Ying, and J. Leskovec (2017) Inductive representation learning on large graphs. In Proceedings of the Advances in Neural Information Processing Systems, pp. 1024–1034. Cited by: §5.1, §5.2, §5.2.
  • K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    ,
    pp. 770–778. Cited by: §5.2.
  • S. Ioffe and C. Szegedy (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In

    Proceedings of the International Conference on Machine Learning

    ,
    pp. 448–456. Cited by: §1.1, §5.1.
  • M. Kim, D. Lee, and H. Shin (2019) Semi-supervised learning for hierarchically structured networks. Pattern Recognition 95, pp. 191 – 200. External Links: ISSN 0031-3203 Cited by: §1.
  • D. P. Kingma and J. Ba (2015) Adam: a method for stochastic optimization. See DBLP:conf/iclr/2015, Cited by: §5.2.
  • T. N. Kipf and M. Welling (2017) Semi-supervised classification with graph convolutional networks. In Proceedings of the International Conference on Learning Representations, Cited by: §1, §5.2, §5.2.
  • B. Knyazev, G. W. Taylor, and M. Amer (2019) Understanding attention and generalization in graph neural networks. In Proceedings of the Advances in Neural Information Processing Systems, pp. 4204–4214. Cited by: §1.1, §2.3, §4.2.1, §4.3.
  • J. B. Lee, R. A. Rossi, S. Kim, N. K. Ahmed, and E. Koh (2019) Attention models in graphs: a survey. ACM Transactions on Knowledge Discovery from Data 13 (6), pp. 62. Cited by: §1.
  • J. B. Lee, R. Rossi, and X. Kong (2018) Graph classification using structural attention. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1666–1674. Cited by: §2.2.
  • X. Li, X. Yan, Q. Gu, H. Zhou, D. Wu, and J. Xu (2019a) DeepChemStable: chemical stability prediction with an attention-based graph convolution network. Journal of Chemical Information and Modeling 59 (3), pp. 1044–1049. Cited by: §1.
  • Y. Li, L. Guo, and Z. Zhou (2019b) Towards safe weakly supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (), pp. 1–1. External Links: Document, ISSN 1939-3539 Cited by: §1.
  • Z. Liu, W. Liu, P. Chen, C. Zhuang, and C. Song (2019) HpGAT: high-order proximity informed graph attention network. IEEE Access 7, pp. 123002–123012. Cited by: §1.
  • T. Nguyen and T. Ho (2012) Detecting disease genes based on semi-supervised learning and protein–protein interaction networks. Artificial Intelligence in Medicine 54 (1), pp. 63–71. Cited by: §1.
  • P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Galligher, and T. Eliassi-Rad (2008) Collective classification in network data. AI Magazine 29 (3), pp. 93–93. Cited by: §5.1.
  • U. S. Shanthamallu, J. J. Thiagarajan, H. Song, and A. Spanias (2019) GrAMME: semisupervised learning using multilayered graph attention models. IEEE Transactions on Neural Networks and Learning Systems (), pp. 1–12. External Links: Document, ISSN 2162-2388 Cited by: §2.2.
  • P. Shaw, J. Uszkoreit, and A. Vaswani (2018) Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 464–468. Cited by: §2.2.
  • Y. Shi, M. Lei, H. Yang, and L. Niu (2019) Diffusion network embedding. Pattern Recognition 88, pp. 518–531. Cited by: §2.1.
  • N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov (2014) Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15 (1), pp. 1929–1958. Cited by: §5.2.
  • S. Vashishth, P. Yadav, M. Bhandari, and P. Talukdar (2019) Confidence-based graph convolutional networks for semi-supervised learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pp. 1792–1801. Cited by: §2.1, §5.2, §5.2.
  • A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, pp. 5998–6008. Cited by: §2.2, §3.
  • P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio (2018) Graph attention networks. In Proceedings of the International Conference on Learning Representations, Cited by: §1.1, §1, §1, §5.1.
  • Y. Wang, Y. Yao, H. Tong, F. Xu, and J. Lu (2018) A brief review of network embedding. Big Data Mining and Analytics 2 (1), pp. 35–47. Cited by: §2.2.
  • B. Wu, Y. Liu, B. Lang, and L. Huang (2018)

    Dgcnn: disordered graph convolutional neural network based on the gaussian mixture model

    .
    Neurocomputing 321, pp. 346–356. Cited by: §1.
  • Z. Xiong, D. Wang, X. Liu, F. Zhong, X. Wan, X. Li, Z. Li, X. Luo, K. Chen, H. Jiang, and M. Zheng (2019) Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of Medicinal Chemistry. Note: PMID: 31408336 External Links: Document Cited by: §1.
  • K. Xu, W. Hu, J. Leskovec, and S. Jegelka (2019) How powerful are graph neural networks?. In Proceedings of the International Conference on Learning Representations, Cited by: §2.1.
  • Z. Yang, W. Cohen, and R. Salakhudinov (2016) Revisiting semi-supervised learning with graph embeddings. In Proceedings of the International Conference on Machine Learning, pp. 40–48. Cited by: §1, §2.1.
  • X. Yue, Z. Wang, J. Huang, S. Parthasarathy, S. Moosavinasab, Y. Huang, S. M. Lin, W. Zhang, P. Zhang, and H. Sun (2019) Graph embedding on biomedical networks: methods, applications and evaluations. Bioinformatics. Note: btz718 External Links: ISSN 1367-4803, Document Cited by: §1.
  • Q. Zhang, J. Chang, G. Meng, S. Xu, S. Xiang, and C. Pan (2019a) Learning graph structure via graph convolutional networks. Pattern Recognition 95, pp. 308–318. Cited by: §1.
  • S. Zhang, H. Tong, J. Xu, and R. Maciejewski (2018) Graph convolutional networks: algorithms, applications and open challenges. In Proceedings of the International Conference on Computational Social Networks, pp. 79–91. Cited by: §1.
  • Z. Zhang, D. Chen, J. Wang, L. Bai, and E. R. Hancock (2019b) Quantum-based subgraph convolutional neural networks. Pattern Recognition 88, pp. 38–49. Cited by: §1.
  • Z. Zhang, D. Chen, Z. Wang, H. Li, L. Bai, and E. R. Hancock (2019c) Depth-based subgraph convolutional auto-encoder for network representation learning. Pattern Recognition 90, pp. 363–376. Cited by: §1.