1 Introduction and related work
The number of articles published every year keeps increasing Druss and Marcus (2005); Larsen and Von Ins (2010) and well over 50 million scholarly articles have been published so far Jinha (2010). While this repository of human knowledge contains invaluable information, it has become increasingly difficult to take advantage of all available information due to its sheer amount.
One challenge is that the knowledge present in scholarly articles is mostly unstructured. One approach to organize this knowledge is to classify each sentenceKim et al. (2011); Amini et al. (2012); Hassanzadeh et al. (2014); Dernoncourt et al. (2016). Another approach is to extract entities and relations between them, which is the focus of the ScienceIE shared task at SemEval-2017 Augenstein et al. (2017).
Relation extraction can be seen as a process comprising two steps that can be done jointly Li and Ji (2014) or separately: first, entities of interest need to be identified, then the relation among each possible set of entities has to be determined. In this work, we concentrate on the second step (often referred to as relation extraction or classification) and on binary relations, i.e. relations between two entities. Extracted relations can be used for a variety of tasks such as question-answering systems Ravichandran and Hovy (2002), ontology extension Schutz and Buitelaar (2005), and clinical trials Frunza and Inkpen (2011).
In this paper, we describe the system that we submitted for the ScienceIE shared task. Our system is based on convolutional neural networks and ranked first for relation extraction (subtask C).
|Examples||Rule format||Relations detected|
|transmission electron microscopy (TEM)||A (B)||If B is an abbreviation of A, then A and B are synonyms.|
|high purity standard metals (Sn, Pb, Zn, Al, Ag, Ni)||A (B, C, … , D)||If any of B, C, … , D is a hyponym of A, then all of them are hyponyms of A.|
|(TEMs), scanning electron microscopes||(A) B||A and B have no relation.|
|DOTMA/DOPE||A/B||A and B have no relation.|
Existing systems for relation extraction can be classified into five categories Zettlemoyer (2013): systems based on hand-built patterns Yangarber and Grishman (1998), bootstrapping methods Brin (1998), unsupervised methods Gonzalez and Turmo (2009), distant supervision Snow et al. (2004), and supervised methods. We focus on supervised methods, as the ScienceIE shared task provides a labeled training set.
Supervised methods for relation extraction commonly employ support vector machinesUzuner et al. (2010, 2011); Minard et al. (2011); GuoDong et al. (2005), naïve Bayes Zayaraz and Kumara (2015), maximum entropy Sun and Grishman (2012), or conditional random fields Sutton and McCallum (2006). These methods require the practitioner to handcraft features, such as surface features, lexical features, syntactic features Grouin et al. (2010) or features derived from existing ontologies Rink et al. (2011). The use of kernels based on dependency trees has also been explored Bunescu and Mooney (2005); Culotta and Sorensen (2004); Zhou et al. (2007).
Our model for relation extraction comprises three parts: preprocessing, convolutional neural network (CNN), and postprocessing.
The preprocessing step takes as input each raw text (i.e., in ScienceIE, a paragraph of a scientific article) as well as the location of all entities present in the text, and output several examples. Each example is represented as a list of tokens, each with four features: the relative positions of the two entity mentions, and their entity types and part-of-speech (POS) tags. Figure 1 shows an example from the ScienceIE corpus in the table on the left.
Sentence and token boundaries as well as POS tags are detected using the Stanford CoreNLP toolkit Manning et al. (2014), and every pair of entity mentions of the same type within each sentence boundary are considered to be of a potential relation. We also remove any references (e.g. [1, 2]) that are irrelevant to the task, and ensure that the sentences are not too long by eliminating the tokens before the beginning of the first entity mention and after the end of the second entity mention.
2.2 CNN architecture
Second, the CNN takes each preprocessed sentence as input, and predicts the relation between the two entities. The CNN architecture, illustrated in Figure 1, consists of four main layers, similar to the one used in text classification Collobert et al. (2011); Kim (2014); Lee and Dernoncourt (2016); Gehrmann et al. (2017).
the embedding layer converts each feature (word, relative positions 1 / 2, type of entity, and POS tag) into an embedding vector via a lookup table and concatenates them.
the convolutional layer
with ReLU activation transforms the embeddings into feature maps by sliding filters over the tokens.
the max pooling layer takes the most effective feature in each feature map by applying the max operator.
the fully connected layer
with softmax activation outputs the probability of each relation.
|annotation||A (arg1) is a Hyponym of (rel) B (arg2)|
|order in text||… A … B …||… B … A …|
|w/ neg. smpl.||None||B||A||None||B||A|
|annotation||A (arg) is a Synonym of (rel) B (arg)|
|order in text||… A … B …||… B … A …|
|w/ neg. smpl.||Syn||B||A||Syn||B||A|
2.3 Rule-based postprocessing
Finally, the postprocessing step uses the rules in Table 1 to correct the relations detected by the CNN, or to detect additional relations. These rules were developed from the examples in the training set, to be consistent with common sense.
During training, the objective is to maximize the log probability of the correct relation type. The model is trained using stochastic gradient descent with minibatch of size 16, updating all parameters, i.e., token embeddings, feature embeddings, CNN filter weights, and fully connected layer weights, at each gradient descent step. For regularization, dropout is applied before the fully connected layer, and early stop with a patience of 10 epochs is used based on the development set.
The token embeddings are initialized using publicly available111http://nlp.stanford.edu/projects/glove/ pre-trained token embeddings, namely GloVe Pennington et al. (2014) trained on Wikipedia and Gigaword 5 Parker et al. (2011). The feature embeddings and the other parameters of the neural network are initialized randomly.
To deal with class imbalance, we upsampled the synonym and hyponym classes by duplicating the examples in the positive classes so that the upsampling ratio, i.e., the ratio of the number of positive examples in each class to that of the negative examples, is at least 0.5. Without the upsampling, it was impossible to train the model.
We evaluate our model on the ScienceIE dataset Augenstein et al. (2017), which consists of 500 journal articles evenly distributed among the domains of computer science, material sciences and physics. Three types of entities are annotated: process, task, and material. The relation between each pair of entity of the same type within a sentence are annotated as either “Synonym-of”, “Hyponym-of”, or “None”. Table 3 shows the number of examples for each relation class.
details the experiment ranges and choices of hyperparameters. The results were quite robust to the choice of hyperparameters within the specified ranges.
|Token embedding dim.||100||50 – 300|
|Feature embedding dim.||10||5 – 50|
|CNN filter height||5||3 – 15|
|Number of CNN filters||200||50 – 500|
|Dropout probability||0.5||0 – 1|
|Upsampling ratio||3||0.5 – 5|
3.3 Argument ordering strategies
|All||correct order||any order||0.193||0.101||0.132||0.782||0.640||0.703||0.409||0.245||0.306|
|corr. w/ n. s.||any order||0.431||0.127||0.196||0.826||0.756||0.788||0.638||0.295||0.404|
|any order||any order||0.482||0.197||0.279||0.784||0.756||0.769||0.621||0.346||0.444|
|any order||fixed order||0.486||0.195||0.278||0.773||0.753||0.763||0.621||0.345||0.443|
|fixed order||any order||0.372||0.218||0.274||0.743||0.756||0.749||0.516||0.362||0.425|
|fixed order||fixed order||0.425||0.213||0.283||0.803||0.753||0.777||0.578||0.358||0.441|
|Hyponym||correct order||any order||0.108||0.069||0.084||-||-||-||-||-||-|
|corr. w/ n. s.||any order||0.215||0.115||0.148||-||-||-||-||-||-|
|any order||any order||0.384||0.246||0.299||-||-||-||-||-||-|
|any order||fixed order||0.410||0.235||0.298||-||-||-||-||-||-|
|fixed order||any order||0.385||0.249||0.301||-||-||-||-||-||-|
|fixed order||fixed order||0.409||0.237||0.297||-||-||-||-||-||-|
|Synonym||any order||any order||-||-||-||0.855||0.771||0.811||-||-||-|
|any order||fixed order||-||-||-||0.852||0.776||0.812||-||-||-|
|Hyp+Syn||any + any||any + fixed||0.385||0.228||0.285||0.857||0.771||0.812||0.553||0.373||0.445|
One of the main challenges in relation extraction is the ordering of arguments in relations, as many relations are order-sensitive. For example, consider the sentence “A dog is an animal.” If we set “dog” be the first argument and “animal” the second, then the corresponding relation is “Hyponym-of”; however, if we reverse the argument order, then the “Hyponym-of” relation does not hold any more.
Therefore, it is crucial to ensure that 1) the CNN is provided with the information about the argument order, and 2) it is able to utilize the given information efficiently. In our work, the former point is addressed by providing the CNN with the two relative position features compared to the first and the second argument of the relation respectively. In order to certify the latter point, we experimented with four strategies for argument ordering, outlined in Table 2.
4 Results and Discussion
Table 3 shows the results from experimenting with various argument ordering strategies. The correct order strategy performed the worst, but the negative sampling improved over it slightly, while the fixed order and any order strategies performed the best. The latter two strategies performed almost equally well in terms of micro-averaged F1-score. This implies that for relation extraction it may be advantageous to use both the original relation classes as well as their “reverse” relation classes for training, instead of using only the original relation classes with the “correct” argument ordering (with or without the negative sampling). Moreover, ordering the argument as the order of appearance in the text and training once per relation (i.e., fixed order) is as efficient as training each relation as two examples in two possible argument ordering, one with the original relation class and the other with the reverse relation class (i.e., any order), despite the small size of the dataset.
The difference in performance between the correct order versus the fixed or any order strategies is more prominent for the “Hyponym-of” relation than for the “Synonym-of” relation. This is expected, since the argument ordering strategy is different only for the order-sensitive “Hyponym-of” relation. It is somewhat surprising though, that the correct order strategy performs worse then the other strategies even for order-insensitive “Synonym-of” relation. This may be due to the fact that the model does not see any training examples with the reversed argument ordering for the “Synonym-of” relation. In comparison, the negative sampling strategy, which learns from both the original and reversed argument ordering for the “Synonym-of” relation, the performance is comparable to the two best performing strategies.
We have also experimented with different evaluation strategies for the models trained with the any order and fixed order strategies. When the model is trained with the any order strategy, the choice of the evaluation strategy does not impact the performance. In contrast, when the model is trained with the fixed order strategy, it performs better if the same strategy is used for evaluation. This may be the reason that the model trained with the correct order strategy does not perform as well, since it has to be evaluated with a different strategy from training, namely the any order strategy, as we do not know the correct ordering of arguments for examples in the test set.
We have also tried training binary classifiers for the “Hyponym-of” and the “Synonym-of” relations separately and then merging the outputs of the best classifiers for each relations. While the binary classifiers individually performed better than the multi-way classifier for each corresponding relation class, the overall performance based on the micro-averaged F1-score did not improve over the multi-way classifier after merging the outputs of the hyponym and the synonym classifiers.
Based on the results from the argument ordering strategy experiments, we submitted the model trained using the fixed order strategy, which ranked number one in the challenge. The result is shown in Table 6.
To quantify the importance of various features of our model, we trained the model by gradually adding more features one by one, from word embeddings, relative positions, and entity types to POS tags in order. The results on the importance of the features as well as postprocessing are shown in Figure 2. Adding the relative position features improved the performance the most, while adding the entity type improved it the least.
Figure 3 quantifies the impact of two preprocessing steps, deleting brackets and cutting sentences, introduced to compensate for the small dataset size. Cutting the sentence before the first entity and after the second entity resulted in a dramatic impact on the performance, while deleting brackets (i.e., removing the reference marks) improve the performance modestly. This implies that the text between the two entities contains most of the information about the relation between them.
In this article we have presented an ANN-based approach to relation extraction, which ranked first in the SemEval-2017 task 10 (ScienceIE) for relation extraction in scientific articles (subtask C). We have experimented with various strategies to incorporate argument ordering for ordering-sensitive relations, showing that an efficient strategy is to fix the arguments ordering as appears on the text by introducing reverse relations. We have also demonstrated that cutting the sentence before the first entity and after the second entity is effective for small datasets.
The authors would like to thank the Science IE organizers for managing this shared task. The project was supported by Philips Research. The content is solely the responsibility of the authors and does not necessarily represent the official views of Philips Research.
- Amini et al. (2012) Iman Amini, David Martinez, and Diego Molla. 2012. Overview of the ALTA 2012 Shared Task. In Australasian Language Technology Association Workshop 2012. volume 7, page 124.
- Augenstein et al. (2017) Isabelle Augenstein, Mrinal Kanti Das, Sebastian Riedel, Lakshmi Nair Vikraman, and Andrew McCallum. 2017. SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications. In Proceedings of the International Workshop on Semantic Evaluation. Association for Computational Linguistics, Vancouver, Canada.
- Brin (1998) Sergey Brin. 1998. Extracting patterns and relations from the world wide web. In International Workshop on The World Wide Web and Databases. Springer, pages 172–183.
Bunescu and Mooney (2005)
Razvan C Bunescu and Raymond J Mooney. 2005.
A shortest path dependency kernel for relation extraction.
Proceedings of the conference on human language technology and empirical methods in natural language processing. Association for Computational Linguistics, pages 724–731.
Collobert et al. (2011)
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray
Kavukcuoglu, and Pavel Kuksa. 2011.
Natural language processing (almost) from scratch.
The Journal of Machine Learning Research12:2493–2537.
- Culotta and Sorensen (2004) Aron Culotta and Jeffrey Sorensen. 2004. Dependency tree kernels for relation extraction. In Proceedings of the 42nd annual meeting on association for computational linguistics. Association for Computational Linguistics, page 423.
- Dernoncourt et al. (2016) Franck Dernoncourt, Ji Young Lee, and Peter Szolovits. 2016. Neural networks for joint sentence classification in medical paper abstracts. EACL 2017 .
- Druss and Marcus (2005) Benjamin G Druss and Steven C Marcus. 2005. Growth and decentralization of the medical literature: implications for evidence-based medicine. Journal of the Medical Library Association 93(4):499.
Frunza and Inkpen (2011)
Oana Frunza and Diana Inkpen. 2011.
Extracting relations between diseases, treatments, and tests from
Canadian Conference on Artificial Intelligence. Springer, pages 140–145.
- Gehrmann et al. (2017) Sebastian Gehrmann, Franck Dernoncourt, Yeran Li, Eric T. Carlson, Joy T. Wu, Jonathan Welt, David W. Grant, Patrick D. Tyler, and Leo A. Celi. 2017. Comparing rule-based and deep learning models for patient phenotyping. arXiv:1703.08705 .
- Gonzalez and Turmo (2009) Edgar Gonzalez and Jordi Turmo. 2009. Unsupervised relation extraction by massive clustering. In Data Mining, 2009. ICDM’09. Ninth IEEE International Conference on. IEEE, pages 782–787.
- Grouin et al. (2010) Cyril Grouin, Asma Ben Abacha, Delphine Bernhard, Bruno Cartoni, Louise Deleger, Brigitte Grau, Anne-Laure Ligozat, Anne-Lyse Minard, Sophie Rosset, and Pierre Zweigenbaum. 2010. Caramba: concept, assertion, and relation annotation using machine-learning based approaches. In i2b2 Medication Extraction Challenge Workshop.
- GuoDong et al. (2005) Zhou GuoDong, Su Jian, Zhang Jie, and Zhang Min. 2005. Exploring various knowledge in relation extraction. In Proceedings of the 43rd annual meeting on association for computational linguistics. Association for Computational Linguistics, pages 427–434.
- Hashimoto et al. (2013) Kazuma Hashimoto, Makoto Miwa, Yoshimasa Tsuruoka, and Takashi Chikayama. 2013. Simple customization of recursive neural networks for semantic relation classification. In EMNLP. pages 1372–1376.
- Hassanzadeh et al. (2014) Hamed Hassanzadeh, Tudor Groza, and Jane Hunter. 2014. Identifying scientific artefacts in biomedical literature: The evidence based medicine use case. Journal of biomedical informatics 49:159–170.
Arif E Jinha. 2010.
Article 50 million: an estimate of the number of scholarly articles in existence.Learned Publishing 23(3):258–263.
- Kim et al. (2011) Su Nam Kim, David Martinez, Lawrence Cavedon, and Lars Yencken. 2011. Automatic classification of sentences to support evidence based medicine. BioMed Central (BMC) Bioinformatics 12(2):1.
- Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics (ACL), pages 1746–1751.
- Larsen and Von Ins (2010) Peder Olesen Larsen and Markus Von Ins. 2010. The rate of growth in scientific publication and the decline in coverage provided by science citation index. Scientometrics 84(3):575–603.
- Lee and Dernoncourt (2016) Ji Young Lee and Franck Dernoncourt. 2016. Sequential short-text classification with recurrent and convolutional neural networks. In Human Language Technologies 2016: The Conference of the North American Chapter of the Association for Computational Linguistics, NAACL HLT.
- Li and Ji (2014) Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In ACL (1). pages 402–412.
- Manning et al. (2014) Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations. pages 55–60. http://www.aclweb.org/anthology/P/P14/P14-5010.
- Minard et al. (2011) Anne-Lyse Minard, Anne-Laure Ligozat, and Brigitte Grau. 2011. Multi-class svm for relation extraction from clinical reports. In Ranlp. volume 59, pages 604–609.
- Nguyen and Grishman (2015) Thien Huu Nguyen and Ralph Grishman. 2015. Relation extraction: Perspective from convolutional neural networks. In Proceedings of NAACL-HLT. pages 39–48.
- Parker et al. (2011) Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English Gigaword fifth edition. Technical report, Linguistic Data Consortium, Philadelphia.
- Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: global vectors for word representation. Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014) 12:1532–1543.
- Ravichandran and Hovy (2002) Deepak Ravichandran and Eduard Hovy. 2002. Learning surface text patterns for a question answering system. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pages 41–47.
- Rink et al. (2011) Bryan Rink, Sanda Harabagiu, and Kirk Roberts. 2011. Automatic extraction of relations between medical concepts in clinical texts. Journal of the American Medical Informatics Association 18(5):594–600.
- Schutz and Buitelaar (2005) Alexander Schutz and Paul Buitelaar. 2005. Relext: A tool for relation extraction from text in ontology extension. In International semantic web conference. Springer, volume 2005, pages 593–606.
- Snow et al. (2004) Rion Snow, Daniel Jurafsky, Andrew Y Ng, et al. 2004. Learning syntactic patterns for automatic hypernym discovery. In NIPS. volume 17, pages 1297–1304.
- Socher et al. (2012) Richard Socher, Brody Huval, Christopher D Manning, and Andrew Y Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguistics, pages 1201–1211.
- Sun and Grishman (2012) Ang Sun and Ralph Grishman. 2012. Active learning for relation type extension with local and global data views. In Proceedings of the 21st ACM international conference on Information and knowledge management. ACM, pages 1105–1112.
- Sutton and McCallum (2006) Charles Sutton and Andrew McCallum. 2006. An introduction to conditional random fields for relational learning. Introduction to statistical relational learning pages 93–128.
- Uzuner et al. (2010) Ozlem Uzuner, Jonathan Mailoa, Russell Ryan, and Tawanda Sibanda. 2010. Semantic relations for problem-oriented medical records. Artificial intelligence in medicine 50(2):63–73.
- Uzuner et al. (2011) Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association 18(5):552–556.
- Xu et al. (2015) Kun Xu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2015. Semantic relation classification via convolutional neural networks with simple negative sampling. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 536–540. https://aclweb.org/anthology/D/D15/D15-1062.
- Yangarber and Grishman (1998) Roman Yangarber and Ralph Grishman. 1998. Nyu: Description of the proteus/pet system as used for muc-7. In In Proceedings of the Seventh Message Understanding Conference (MUC-7). Citeseer.
- Zayaraz and Kumara (2015) Godandapani Zayaraz and Suresh Kumara. 2015. Concept relation extraction using naïve bayes classifier for ontology-based question answering systems. Journal of King Saud University-Computer and Information Sciences 27(1):13–24.
- Zettlemoyer (2013) Luke Zettlemoyer. 2013. Relation extraction. CSE517: Natural Language Processing .
- Zhou et al. (2007) Guodong Zhou, Min Zhang, Dong-Hong Ji, and Qiaoming Zhu. 2007. Tree kernel-based relation extraction with context-sensitive structured parse tree information. In EMNLP-CoNLL. Citeseer, volume 2007, pages 728–736.