SemEval 2018 Task 4 is an entity linking task on multiparty dialogue.111https://competitions.codalab.org/competitions/17310 It consists in predicting the referents of nominals that refer to a person, such as she, mom, Judy – henceforth mentions. The set of possible referents is given beforehand, as well as the set of mentions to resolve. The dataset used in this task is based on Chen and Choi (2016) and Chen et al. (2017), and consists of dialogue from the TV show Friends in textual form.
Our main interest is whether deep learning models for tasks like entity linking can benefit from having an explicitentity library
, i.e., a component of the neural network that stores entity representations learned during training. To that end, we add such a component to an otherwise relatively basic model – a bidirectional LSTM (long short-term memory;Hochreiter and Schmidhuber 1997), the standard neural network model for sequential data like language. Training and evaluating this model on the task shows that the entity library is beneficial in the case of infrequent entities.222Source code for our model and for the training procedure is published on https://github.com/amore-upf/semeval2018-task4.
2 Related Work
Previous entity linking tasks concentrate on linking mentions to Wikipedia pages (Bunescu and Paşca 2006; Mihalcea and Csomai 2007 and much subsequent work; for a recent approach see Francis-Landau et al. 2016). By contrast, in the present task (based on Chen and Choi 2016; Chen et al. 2017) only a list of entities is given, without any associated encyclopedic entries. This makes the task more similar to the way in which a human audience might watch the TV show, in that they are initially unfamiliar with the characters. What also sets the present task apart from most previous tasks is its focus on multiparty dialogue (as opposed to, typically, newswire articles).
A task that is closely related to entity linking is coreference resolution, i.e., the task of clustering mentions that refer to the same entity (e.g., the CoNLL shared task of Pradhan et al. 2011). Since mention clusters essentially correspond to entities (an insight central to the approaches to coreference in Haghighi and Klein 2010; Clark and Manning 2016), the present task can be regarded as a type of coreference resolution, but one where the set of referents to choose from is given beforehand.
Since our main aim is to test the benefits of having an entity library, in other respects our model is kept more basic than existing work both on entity linking and on coreference resolution (e.g., the aforementioned approaches, as well as Wiseman et al. 2016; Lee et al. 2017, Francis-Landau et al. 2016). For instance, we avoid feature engineering, focusing instead on the model’s ability to learn meaningful entity representations from the dialogue itself. Moreover, we deviate from the common strategy to entity linking of incorporating a specialized coreference resolution module (e.g., Chen et al. 2017).
3 Model Description
We approach the task of character identification as one of multi-class classification. Our model is depicted in Figure 1, with inputs in the top left and outputs at the bottom. In a nutshell, our model is a bidirectional LSTM (long short-term memory, Hochreiter and Schmidhuber 1997
) that processes the dialogue text and resolves mentions, through a comparison between the LSTM’s hidden state (for each mention) to vectors in a learned entity library.
The model is given chunks of dialogue, which it processes token by token. The th token and its speakers (typically a singleton set) are represented as one-hot vectors, embedded via two distinct embedding matrices ( and , respectively) and finally concatenated to form a vector (Eq. 1; see also Figure 1). In case contains multiple speakers, their embeddings are summed.
We apply an activation function. The hidden state of a unidirectional LSTM for the th input is recursively defined as a combination of that input with the LSTM’s previous hidden state . For a bidirectional LSTM, the hidden state is a concatenation of the hidden states and of two unidirectional LSTMs which process the data in opposite directions (Eq. 2; see also Figure 1). In principle, this enables a bidirectional LSTM to represent the entire dialogue with a focus on the current input, including for instance its relevant dependencies on the context.
In the model, learned representations of each entity are stored in the entity library (see Figure 1): is a matrix which represents each of entities through a -dimensional vector, and whose values are updated (only) during training. For every token that is tagged as a mention,333For multi-word mentions this is done only for the last token in the mention. we map the corresponding hidden state to a vector . This extracted representation is used to retrieve the (candidate) referent of the mention from the entity library: The similarity of to each entity representation stored in
is computed using cosine, and softmax is then applied to the resulting similarity profile to obtain a probability distributionover entities (‘class scores’ in Figure 1):
At testing time, the model’s prediction for the th token is the entity with highest probability:
4 Experimental Setup
We use the training and test data provided for SemEval 2018 Task 4, which span the first two seasons of the TV show Friends, divided into scenes (train: scenes from episodes; test: scenes from episodes).444The organizers also provided data divided by episodes rather than scenes, which we didn’t use. In total, the training and test data contain 13,280 and 2,429 nominal mentions (e.g., Ross, I; Figure 2), respectively, which are annotated with the ID of the entity to which they refer (e.g., 335, 183). The utterances are further annotated with the name of the speaker (e.g., Joey Tribbiani). Overall there are 372 entities in the training data (test data: 106).
Our models do not use any of the provided automatic linguistic annotations, such as PoS or named entity tags.
Using 5-fold cross-validation on the training data, we performed a random search over the hyperparameters and chose those which yielded the best mean F1-score. Specifically, our submitted model is trained in batch mode using the Adam optimizerKingma and Ba (2014) with a learning rate of . Each batch covers scenes, which are given to the model in chunks of tokens. The token embeddings () are initialized with the word2vec vectors. Dropout rates of and are applied on the input and hidden layer of the LSTM, respectively. The size of is set to units, the embeddings of the entity library and speakers are set to dimensions.
Other configurations, including randomly initialized token embeddings, weight sharing between and , self-attention Bahdanau et al. (2016) on the input layer, a uni-directional LSTM, and rectifier or linear activation function on the input embeddings did not improve performance.
For the final submission of the answers for the test data, we created an ensemble model by averaging the output (Eq. 3) of the five models trained on the different folds.
|all entities||main entities|
Two evaluation conditions were defined by the organizers – all entities and main entities – with macro-average F-score and label accuracy as the official metrics, and macro-average F-score in the all entities condition applied to the leaderboard. The all entities evaluation has 67 classes: 66 for entities that are mentioned at least 3 times in the test set and one grouping all others. The main entities evaluation has 7 classes, 6 for the main characters and one for all the others. Among all four participating systems in this SemEval task our model achieved the highest score on the all entities evaluation, and second-highest on the main entities evaluation.
Table 1 gives our results in the two evaluations, comparing the models described in Section 4. While both models perform on a par on main entities, AMORE-UPF outperforms NoEntLib by a substantial margin when all characters are to be predicted (+15 points in F-score, +3 points in accuracy; Table 1).666The mean difference between the single models (trained on a single fold) and the ensemble AMORE-UPF is between -1.3 points (accuracy main entities, std ) and -4.2 points (F-score all entities, std ). The difference between the models with/without an entity library are statistically significant based on approximate randomization tests Noreen (1989), with the significance level . This shows that the use of an entity library can be beneficial for the linking of rarely mentioned characters.
Figure 3 shows that most of the target mentions in the test data fall into one of five grammatical categories. The dataset contains mostly pronouns (83%), with a very high percentage of first person pronouns (44%). Figures 4 and 5 present the accuracy and F-score which the two models described above obtain on all entities for different categories of mentions. The entity library is beneficial when the mention is a first person pronoun or a proper noun (with an increase of 30 points in F-score for both categories; Figure 4), and closer inspection revealed that this effect was larger for rare entities.
The AMORE-UPF model consists of a bidirectional LSTM linked to an entity library. Compared to an LSTM without entity library, NoEntLib, the AMORE-UPF model performs particularly well on rare entities, which explains its top score in the all entities condition of SemEval 2018 Task 4. This finding is encouraging, since rare entities are especially challenging for the usual approaches in NLP, due to the scarcity of information about them.
We offer the following explanation for this beneficial effect of the entity library, as a hypothesis for future work. Having an entity library requires the LSTM of our model to output some representation of the mentioned entity, as opposed to outputting class scores more or less directly as in the variant NoEntLib. Outputting a meaningful entity representation is particularly easy in the case of first person pronouns and nominal mentions (where the effect of the entity library appears to reside; Figure 4): the LSTM can learn to simply forward the speaker embedding unchanged in the case of pronoun I, and the token embedding in the case of nominal mentions. This strategy does not discriminate between frequent and rare entities; it works for both alike. We leave further analyses required to test this potential explanation for future work.
Future work may also reveal to what extent the induced entity representations may be useful in others, to what extent they encode entities’ attributes and relations (cf. Gupta et al. 2015
), and to what extent a module like our entity library can be employed elsewhere, in natural language processing and beyond.
This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 715154), and from the Spanish Ramón y Cajal programme (grant RYC-2015-18907). We are grateful to the NVIDIA Corporation for the donation of GPUs used for this research. We are also very grateful to the Pytorch developers. This paper reflects the authors’ view only, and the EU is not responsible for any use that may be made of the information it contains.
- Bahdanau et al. (2016) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2016. Neural Machine Translation by Jointly Learning to Align and Translate. Computing Research Repository, abs/1409.0473v7.
Boleda et al. (2017)
Gemma Boleda, Sebastian Padó, Nghia The Pham, and Marco Baroni. 2017.Living a Discrete Life in a Continuous World: Reference in Cross-modal Entity Tracking. In IWCS 2017 — 12th International Conference on Computational Semantics — Short papers.
- Bunescu and Paşca (2006) Razvan Bunescu and Marius Paşca. 2006. Using Encyclopedic Knowledge for Named Entity Disambiguation. In 11th conference of the European Chapter of the Association for Computational Linguistics.
- Chen et al. (2017) Henry Y. Chen, Ethan Zhou, and Jinho D. Choi. 2017. Robust Coreference Resolution and Entity Linking on Dialogues: Character Identification on TV Show Transcripts. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 216–225.
- Chen and Choi (2016) Yu-Hsin Chen and Jinho D Choi. 2016. Character Identification on Multiparty Conversation: Identifying Mentions of Characters in TV Shows. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 90–100.
- Clark and Manning (2016) Kevin Clark and Christopher D. Manning. 2016. Improving Coreference Resolution by Learning Entity-Level Distributed Representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653.
- Francis-Landau et al. (2016) Matthew Francis-Landau, Greg Durrett, and Dan Klein. 2016. Capturing Semantic Similarity for Entity Linking with Convolutional Neural Networks. Computing Research Repository, abs/1604.00734.
- Gupta et al. (2015) Abhijeet Gupta, Gemma Boleda, Marco Baroni, and Sebastian Padó. 2015. Distributional Vectors Encode Referential Attributes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 12–21.
- Haghighi and Klein (2010) Aria Haghighi and Dan Klein. 2010. Coreference Resolution in a Modular, Entity-Centered Model. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 385–393. Association for Computational Linguistics.
- Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735–1780.
- Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. Computing Research Repository, abs/1412.6980.
- Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-End Neural Coreference Resolution. Computing Research Repository, abs/1707.07045.
- Mihalcea and Csomai (2007) Rada Mihalcea and Andras Csomai. 2007. Wikify!: Linking Documents to Encyclopedic Knowledge. In Proceedings of the sixteenth ACM Conference on Information and Knowledge Management, pages 233–242.
- Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and Their Compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, pages 3111–3119.
- Noreen (1989) E.W. Noreen. 1989. Computer-Intensive Methods for Testing Hypotheses: An Introduction. Wiley.
- Pradhan et al. (2011) Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–27. Association for Computational Linguistics.
- Sukhbaatar et al. (2015) Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-To-End Memory Networks. arXiv, pages 1–11.
- Wiseman et al. (2016) Sam Wiseman, Alexander M Rush, and Stuart M Shieber. 2016. Learning Global Features for Coreference Resolution. Computing Research Repository, abs/1604.03035.