This work is licenced under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/Classifying relations between two entities in a given context is an important task in natural language processing (NLP). Take the following sentence as an example: “Jewelry and other smaller [valuables] were locked in a [safe] or a closet with a deadbolt.” The marked entities valuables and safe are of relation Content-Container. Relation classification plays a key role in various NLP applications, and has become a hot research topic in recent years.
Nowadays, neural network-based approaches have made significant improvement in relation classification, compared with traditional methods based on either human-designed features [Kambhatla2004, Hendrickx et al.2009] or kernels [Bunescu and Mooney2005, Plank and Moschitti2013]
. For example, CNN and CNN-NG utilize convolutional neural networks (CNNs) for relation classification. SDP-LSTM apply long short term memory (LSTM)-based recurrent neural networks (RNNs) along the shortest dependency path. EnsembleNN build ensembles of gated recurrent unit (GRU)-based RNNs and CNNs.
We have noticed that these neural models are typically designed in shallow architectures, e.g., one layer of CNN or RNN, whereas evidence in the deep learning community suggests that deep architectures are more capable of information integration and abstraction[Graves et al.2013, Hermans and Schrauwen2013, Irsoy and Cardie2014]. A natural question is then whether such deep architectures are beneficial to the relation classification task.
In this paper, we propose the deep recurrent neural networks (DRNNs) to classify relations. The deep RNNs can explore the representation space in different levels of abstraction and granularity. By visualizing how RNN units are related to the ultimate classification, we demonstrate that different layers indeed learn different representations: low-level layers enable sufficient information mix, while high-level layers are more capable of precisely locating the information relevant to the target relation between two entities. Following our previous work [Xu et al.2015b], we leverage the shortest dependency path (SDP, Figure 1) as the backbone of our RNNs.
We further observe that the relationship between two entities are directed. Two sub-paths, separated by entities’ common ancestor, can be mapped to subject-predicate and object-predicate components of a relation. By changing the order of these two sub-paths, we obtain a new data sample with the inversed relationship (Figure 1b). Such data augmentation technique can provide additional data samples without using external data resources.
We evaluated our proposed method on the SemEval-2010 relation classification task. Even if we do not apply data augmentation, the DRNNs model has achieved a high performance of 84.2% -score with a depth of 3, but the performance decreases when the depth is too large. This is because the deep RNN is a large model, which necessitates more data samples for training. Applying data augmentation can alleviate the problem of data sparseness and sustain a deeper RNN to improve the performance to 86.1%. The results show that both our deep networks and the data augmentation strategy have contributed to the relation classification task, and that they are coupled well together for further performance improvement.
2 Related Work
Traditional methods for relation classification mainly fall into two groups: feature-based or kernel-based. The former approaches extract different types of features and feed them into a classifier, e.g., a maximum entropy model [Kambhatla2004]. Various features, including lexical, syntactic, as well as semantic ones, are shown to be useful to relation classification [Hendrickx et al.2009]. By contrast, kernel-based methods do not have explicit feature representations, but require predefined similarity measure of two data samples. SpdKernel design a kernel along the shortest dependency path (SDP) between two entities by observing that the relation strongly relies on SDPs. EmbedTreeK combine structural information and semantic information in a tree kernel.
Neural networks have now become a prevailing technique in this task. RAE design a recursive neural network along the constituency parse tree. CustomRNN, also on the basis of recursive networks, emphasize more on important phrases; chainRNN restrict recursive networks to SDP. In our previous study [Xu et al.2015b], we introduce SDP-based recurrent neural network to classify relations.
CNN, on the other hand, apply CNNs to relation classification. Along this line, RankCNN replace the common softmax loss function with a ranking loss in their CNN model. CNN-NG design a negative sampling method for SDP-based CNNs.
Besides, representative hybrid models of CNNs and recursive/recurrent networks include DepNN and EnsembleNN.
3 The Proposed Methodology
In this section, we describe our methodology in detail. Subsection 3.1 provides an overall picture of our DRNNs model. Subsections 3.2 and 3.3 describe deep recurrent neural networks. The proposed data augmentation technique is introduced in Subsection 3.4. Finally, we present our training objective in Subsection 3.5.
Figure 2 depicts the overall architecture of the DRNNs model. Given a sentence and its dependency parse tree,111 Parsed by the Stanford parser [de Marneffe et al.2006]. we follow our previous work [Xu et al.2015b] and build DRNNs on the shortest dependency path (SDP), which serves as a backbone. In particular, an RNN picks up information along each sub-path, separated by the common ancestor of marked entities. Also, we take advantage of four information channels, namely, word embeddings, POS embeddings, grammatical relation embeddings, and WordNet embeddings.
Different from SDP-LSTM, we design deep RNNs with up to four hidden layers so as to capture information in different levels of abstraction. For each RNN layer, max pooling gathers information from different recurrent nodes. Notice that the four channels (with eight sub-paths) are processed in a similar way. Then all pooling layers are concatenated and fed into a hidden layer for information integration. Finally, we have a softmax output layer for classification.
3.2 Recurrent Neural Networks on Shortest Dependency Path
In this subsection, we introduce a single layer of RNN based on SDP, serving as a building block of our deep architecture.
Compared with a raw word sequence or a whole parse tree, the shortest dependency path (SDP) between two entities has two main advantages. First, it reduces irrelevant information; second, grammatical relations between words focus on the action and agents in a sentence and are naturally suitable for relation classification. Existing studies have demonstrated the effectiveness of SDP [Ebrahimi and Dou2015, Liu et al.2015, Xu et al.2015b, Xu et al.2015a]; details are not repeated here.
Focused on the SDP, an RNN keeps a hidden state vector, changing with the input word at each step accordingly. Concretely, the hidden state , for the -th word in the sub-path, depends on its previous state and the current word’s embedding
. For the simplicity and without loss of generality, we use vanilla recurrent networks with perceptron-like interaction, that is, the input is linearly transformed by a weight matrix and non-linearly squashed by an activation function, i.e.,
where and are weight matrices for the input and recurrent connections, respectively. is a bias term, and is a non-linear activation function ( in our experiment).
3.3 Deep Recurrent Neural Networks
Although an RNN, as described above, is suitable for picking information along a sequence (a subpath in our task) by its iterative nature, the machine learning community suggests that deep architectures may be more capable of information integration, and can capture different levels of abstraction.
A single-layer RNN can be viewed that it is deep along time steps. When unfolded, however, the RNN has only one hidden layer to capture the current input, as well as to retain the information in its previous step. In this sense, single-layer RNNs are actually shallow in information processing [Hermans and Schrauwen2013, Irsoy and Cardie2014].
In the relation classification task, words along SDPs provide information from different perspectives. On the one hand, the marked entities themselves are informative. On the other hand, the entities’ common ancestor (typically verbs) tells how the two entities are related to each other. Such heterogeneous information might necessitate more complex machinery than a single RNN layer.
Following such intuition, we investigate deep RNNs by stacking multiple hidden layers on the top of one another, that is, every layer treats its previous layer as input, and computes its activation similar to Equation 1. Formally, we have
where the subscripts refer to time steps, and superscripts indicate the layer number. To enhance information propagation, we add a “cross” connection for hidden layers () from the lower layer in the previous time step, given by in Equation 2. (See also and arrows in Figure 2).
3.4 Data Augmentation
Neural networks, especially deep ones, are likely to be prone to overfitting. The SemEval-2010 relation classification dataset, we use, comprises only several thousand samples, which may not fully sustain the training of deep RNNs.
To mitigate this problem, we propose a data augmentation technique for relation classification by making use of the directionality of relationships.
The two sub-paths
valuables jewelry locked
locked in closet safe
in Figure 1, for example, can be mapped to the subject-predicate and object- predicate components in the relation Content-Container. If we change the order of these two sub-paths, we obtain
safe closet in locked
locked jewelry valuables
Then the relationship becomes Container-Content, which is exactly the inverse of Content-Container. In this way, we can augment the dataset without using additional resources.
3.5 Training Objective
For each recurrent layer and embedding layer (over each sub-path for each channel), we apply a max pooling layer to gather information. In total, we have 40 pools, which are concatenated and fed to a hidden layer for information integration.
where refers to the inverse of relation . is the number of data samples in the original training set. is the number of weight matrices in DRNNs. is a regularization coefficient, and denotes Frobenius norm of a matrix.
For decoding (predicting the relation of an unseen sample), the data augmentation technique provides new opportunities, because we can use the probability of , , or both. Section 4.3 provides detailed discussion.
describes hyperparameter settings. We discuss the details of data augmentation in Subsection4.3 and the rationale for using RNNs in Subsection 4.4. Subsection 4.5 compares our DRNNs model with other methods in the literature. In Subsection 4.6, we have quantitative and qualitative analysis of how the depth affects our model.
We evaluated our DRNNs model on the SemEval-2010 Task 8 dataset, which is an established benchmark for relation classification [Hendrickx et al.2009]. The dataset contains 8000 sentences for training, and 2717 for testing. We split 800 samples out of the training set for validation.
There are 9 directed relations and an undirected default relation
Other; thus, we have 19 different labels in total. However, the
Other class is not taken into consideration when we compute the official measures.
4.2 Hyperparameter Settings
This subsection presents hyperparameters of our proposed model. We basically followed the settings in our previous work [Xu et al.2015b]. Word embeddings were 200-dimensional, pretrained ourselves using word2vec [Mikolov et al.2013] on the Wikipedia corpus; embeddings in other channels were 50-dimensional initialized randomly. The hidden layers in each channel had the same number of units as their embeddings (either 200 or 50); the penultimate hidden layer was 100-dimensional. An penalty of was also applied as in SDP-LSTM, but we chose the dropout rate by validation with a granularity of 5% for our model variants (with different depths).
We also chose the depth of DRNNs by validation from the set . The 3-layer and 4-layer DRNNs yield the highest performance with and without data augmentation, respectively. Section 4.6 provides both quantitative and qualitative analysis regarding the effect of depth.
We applied mini-batched stochastic gradient descent for optimization, where gradients were computed by standard back-propagation.
4.3 Data Augmentation Details
As mentioned in Section 4.1, the SemEval-2010 Task 8 dataset contains an undirected class Other in addition to 9 directed relations (18 classes). For data augmentation, it is natural that the inversed Other relation is also in the Other class itself. However, if we augment all the relations, we observe a performance degradation of 0.7% (Table 2). We deem the Other class contains mainly noise, and is inimical to our model. Then we conducted another experiment where we only augmented the Other class. The result verifies our conjecture as we obtained an even larger degradation of 1.1% in this setting.
The pilot experiments suggest that we should take into consideration unfavorable noise when performing data augmentation. In this experiment, if we reverse the directed relations only and leave the Other class intact, the performance is improved by a large margin of 1.9%. This shows that our proposed data augmentation technique does help to mitigate the problem of data sparseness, if we carefully rule out the impact of noise.
During validation and testing, we shall decode the target label of an unseen data sample (with two entities and ). Through data augmentation, we are equipped with the probability of in addition to . In our experiment, we tried several settings and chose to use only, because it yields the highest the validation result. We think this is probably because the Other class brings more noise to than , as the Other class is not augmented (and hence asymmetric).
We would like to point out that our data augmentation method is a general technique for relation classification, which is not ad hoc to a specific dataset; that the methodology for dealing with noise is also potentially applicable to other datasets.
4.4 RNNs vs. CNNs
As both RNNs and CNNs are prevailing neural models for NLP, we are curious whether deep architectures are also beneficial to CNNs. We tried a CNN with a sliding window of size 3 based on SDPs, similar to CNN-NG; other settings were as our DRNNs.
The results are shown in Table 2. We observe that a single layer of CNN is also effective, yielding an
-score slightly worse than our RNN. But the deep architecture hurts the performance of CNNs in this task. One plausible explanation is that, when convolution is performed, the beginning and end of a sentence are typically padded with a special symbol or simply zero. However, the shortest dependency path between two entities is usually not very long (4 on average). Hence, sentence boundaries may play a large role in convolution, which makes CNNs vulnerable.
On the contrary, RNNs can deal with sentence boundaries smoothly, and the performance continues to increase with up to 4 hidden layers. (Details are deferred to Subsection 4.6.)
4.5 Overall Performance
|SVM||POS, WordNet, prefixes and other morphological features,||82.2|
|[Hendrickx et al.2009]||depdency parse, Levin classes, PropBank, FanmeNet,|
|NomLex-Plus, Google -gram, paraphrases, TextRunner|
|[Socher et al.2011]||+ POS, NER, WordNet||77.6|
|[Socher et al.2012]||+ POS, NER, WordNet||82.4|
|[Zeng et al.2014]||+ position embeddings, WordNet||82.7|
|Chain CNN||Word embeddings, POS, NER, WordNet||82.7|
|[Ebrahimi and Dou2015]|
|[dos Santos et al.2015]||+ position embeddings||84.1|
|[Yu et al.2014]||+ dependency parsing, NER||83.0|
|[Xu et al.2015b]||Word + POS + GR + WordNet embeddings||83.7|
|DepNN||Word embeddings + WordNet||83.0|
|[Liu et al.2015]||Word embeddings + NER||83.6|
|depLCNN||Word + WordNet + words around nominals||83.7|
|[Xu et al.2015a]||+ negative sampling from NYT dataset||85.6|
|Ensemble Methods||Word+POS+NER+WordNet embeddings, CNNs, RNNs + Stacking||83.4|
|[Nguyen and Grishman2015]||Word+POS+NER+WordNet embeddings, CNNs, RNNs + Voting||84.1|
|DRNNs||Word+POS+GR+WordNet embeddings w/o data augmentation||84.2|
|+ data augmentation||86.1|
Table 3 compares our DRNNs model with previous state-of-the-art methods.222This paper was preprinted on arXiv on 14 Jan 2016. The first entry in the table presents the highest performance achieved by traditional feature-based methods. 2010SVM feed a variety of handcrafted features to the SVM classifier and achieve an -score of 82.2%.
Recent performance improvements on this dataset are mostly achieved with the help of neural networks. In an early study, MVRNN build a recursive network on constituency trees, but achieve a performance worse than 2010SVM. They extend their recursive network with matrix-vector interaction and elevate the -score to 82.4%. chainRNN restrict the recursive network to SDP, which is slightly better than a sentence-wide network. In our previous study [Xu et al.2015b], we introduce recurrent neural networks based on SDP and improve the -score to 83.7%.
In the school of convolution, CNN construct a CNN on the word sequence; they also integrate word position embeddings, which benefit the CNN architecture.
RankCNN propose a similar CNN model, named CR-CNN, by replacing the common softmax cost function with a ranking-based cost function.
By diminishing the impact of the
Other class, they achieve an -score of 84.1%.
CNN-NG design an SDP-based CNN with negative sampling, improving the performance to 85.6%.
FCM propose a Feature-rich Compositional Embedding Model (FCM), which combines unlexicalized linguistic contexts and word embeddings. They do not use neural networks (at least in the usual sense) and achieve an -score of 83.0%.
Our DRNNs model, along with data augmentation, achieves an -score of 86.1%. Even if we do not apply data augmentation, the DRNNs model yields 84.2% -score, which is also the highest score achieved without special treatment to the noisy Other class. The above results show the effectiveness of DRNNs, especially trained with a large (augmented) dataset.
4.6 Analysis of DRNNs’ Depth
[0,r, , Analysis of the depth. ] In this subsection, we analyze the effect of depth in our DRNNs model. We have tested the depth from the set , and plot the results in Figure 4.6. Initially, the performance increases if the depth is larger in both settings with and without augmentation. However, if we do not augment data, the performance peaks when the depth is 3. Provided with augmented training samples, the -score continues to increase with up to 4 layers, and ends up with an -score of 86.1%.
We next investigate how RNN units in different layers are related to the ultimate task of interest. This is accomplished by tracing back information from pooling layers. Noticing that the pooling layer takes maximum value in each dimension, we can compute how much a hidden layer’s units are gathered by pooling for further processing. In this way, we are able to demonstrate the information flow in RNN hidden units. We plot three examples in Figure 3. Here, rectangles refer to RNN hidden layers, unfolded along time. (Rounded rectangles are word embeddings.) The intensity of color reflects the ratio of the pooling proportion.
Sample 1: “Until 1864 vessels in the service of certain UK public offices defaced the Red Ensign with the badge of their office” with label
Instrument-Agency. Its two sub-paths of SDP are
defaced with badge
From Figure 3a, we see that entities like vessels and badge are darker than the verb phrase defaced with on the embedding layer. When information is propagating horizontally and vertically, these entities are getting lighter, while the verb phrase becomes darker gradually. Intuitively, we think that, considering the relation
Instrument-Agency, it is less informative with only two entities vessels and badge. When adding the semantic of verb phrase defaced with, we are more aware of the target relation.333 Using vanilla RNN with a depth of 1, we obtained a slightly better accuracy in this paper than SDP-LSTM.
Sample 2: “Most of the verses of the plantation songs had some reference to freedom” with label
Message-Topic. Its two sub-paths of SDP are
had reference to freedom
Similar to Sample 1, we see from Figure 3b that the color of the “pivot” verb had is getting darker vertically, and becomes the darkest in the fourth RNN layer, indicating the highest pooling portion. This is probably because had links two ends of the relation,
Sample 3: “A more spare, less robust use of classical motifs is evident in a ewer of 1784-85” with label
Component-Whole. Its two sub-paths of SDP are
evident in ewer
Different from Figures 3a and 3b, higher layers pay more attention to entities rather than their ancestor. In this example, motifs and ewer appear to be more relevant to the relation
Wholethan their common ancestor evident. The pooling proportion of entities (motifs, ewer) is increasing, while other words’ proportion is decreasing.
We summarize our findings as follows. (1) Pooled information usually peaks at one or a few words in the embedding layer. This makes sense because there is no information flow in this layer. (2) Information scatters over a wider range in hidden layers, showing that the recurrent propagation does mix information. (3) For a higher-level layer, the network pays more attention to those words that are more relevant to the relation, but whether entities or their common ancestor is more relevant is not consistent among different data samples.
In this paper, we proposed deep recurrent neural networks, named DRNNs, to improve the performance of relation classification. The DRNNs model, consisting of several RNN layers, explores the representation space of different abstraction levels. By visualizing DRNNs’ units, we demonstrated that high-level layers are more capable of integrating information relevant to target relations. In addition, we have designed a data augmentation strategy by leveraging the directionality of relations. When evaluated on the SemEval dataset, our DRNNs model results in substantial performance boost. The performance generally improves when the depth increases; with a depth of 4, our model reaches the highest -measure of 86.1%.
We thank all reviewers for their constructive comments. This research is supported by the National Basic Research Program of China (the 973 Program) under Grant No. 2015CB352201, the National Natural Science Foundation of China under Grant Nos. 61232015, 91318301, 61421091, and 61502014.
- [Bunescu and Mooney2005] Razvan C. Bunescu and Raymond J. Mooney. 2005. A shortest path dependency kernel for relation extraction. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 724–731.
- [de Marneffe et al.2006] Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the International Conference on Language Resources and Evaluation, volume 6, pages 449–454.
- [dos Santos et al.2015] Cıcero Nogueira dos Santos, Bing Xiang, and Bowen Zhou. 2015. Classifying relations by ranking with convolutional neural networks. In Proceedings of 53rd Annual Meeting of the Association for Computational Linguistics, pages 626–634.
- [Ebrahimi and Dou2015] Javid Ebrahimi and Dejing Dou. 2015. Chain based rnn for relation classification. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1244–1249.
- [Graves et al.2013] Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6645–6649.
- [Hashimoto et al.2013] Kazuma Hashimoto, Makoto Miwa, Yoshimasa Tsuruoka, and Takashi Chikayama. 2013. Simple customization of recursive neural networks for semantic relation classification. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1372–1376.
- [Hendrickx et al.2009] Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions, pages 94–99.
- [Hermans and Schrauwen2013] Michiel Hermans and Benjamin Schrauwen. 2013. Training and analysing deep recurrent neural networks. In Advances in Neural Information Processing Systems, pages 190–198.
- [Irsoy and Cardie2014] Ozan Irsoy and Claire Cardie. 2014. Opinion mining with deep recurrent neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 720–728.
- [Kambhatla2004] Nanda Kambhatla. 2004. Combining lexical, syntactic, and semantic features with maximum entropy models for information extraction. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 178–181.
- [Liu et al.2015] Yang Liu, Furu Wei, Sujian Li, Heng Ji, Ming Zhou, and Houfeng WANG. 2015. A dependency-based neural network for relation classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 285–290.
- [Mikolov et al.2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119.
- [Nguyen and Grishman2015] Thien Huu Nguyen and Ralph Grishman. 2015. Combining neural networks and log-linear models to improve relation extraction. arXiv preprint arXiv:1511.05926.
- [Plank and Moschitti2013] Barbara Plank and Alessandro Moschitti. 2013. Embedding semantic similarity in tree kernels for domain adaptation of relation extraction. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1498–1507.
[Socher et al.2011]
Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and
Christopher D. Manning.
Semi-supervised recursive autoencoders for predicting sentiment distributions.In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 151–161.
- [Socher et al.2012] Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211.
- [Xu et al.2015a] Kun Xu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2015a. Semantic relation classification via convolutional neural networks with simple negative sampling. In Proceedings of Conference on Empirical Methods in Natural Language Processing, pages 536–540.
- [Xu et al.2015b] Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. 2015b. Classifying relations via long short term memory networks along shortest dependency paths. In Proceedings of Conference on Empirical Methods in Natural Language Processing, pages 1785–1794.
- [Yu et al.2014] Mo Yu, Matthew Gormley, and Mark Dredze. 2014. Factor-based compositional embedding models. In NIPS Workshop on Learning Semantics.
- [Zeng et al.2014] Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335–2344.