The past years have seen increased interest from the research community in the development of deep reading comprehension models, spurred by the release of datasets such as SQuAD. Rajpurkar et al. (2016). For a majority of these datasets, the top performing models employ span prediction, selecting a span of tokens from the reference document that answers the question. Such models have been very successful; the best model on the SQuAD leaderboard approaches human performance Yu et al. (2018). However, this strong performance may be deceptive. Jia and Liang (2017) show that inserting lexically similar adversarial sentences into the passages sharply reduces performance.
One possible reason for this disparity is that standard span prediction is an easy task. The information required to evaluate whether a span is the correct answer is often located right next to the span. Masala et al. (2017)
transform the SQuAD dataset into a sentence selection task where the goal is to predict the sentence that contains the correct span. They achieve high accuracy on this task using simple heuristics that compare lexical similarity between the question and each sentence individually, without additional context. Selecting an answer from a list of candidate answers that are lexically dissimilar to the context makes it more challenging for models to retrieve the relevant information. For that reason, we focus on reading comprehension for answer selection.
Another common weakness of reading comprehension datasets is that they consist of short paragraphs. This property also makes it easier to locate relevant information from the context. Realistic tasks require answering questions over longer documents.
Building on Clark and Gardner (2017)
, we propose a weighted global normalization method to improve the performance of reading comprehension models for answer selection on long documents. First, we adapt global normalization to the multiple-choice setting by applying a reading comprehension model in parallel over fixed length portions (chunks) of the document and normalizing the scores over all chunks. Global normalization encourages the model to produce low scores when it is not confident that the chunk it is considering contains the information to answer the question. Then we incorporate a weighting function to rescale the contribution of different chunks. In our work we use an Multilayer Perceptron over the scores and a TF-IDF heuristic as our weighting function, but more complex models are possible.
We experiment on the answer selection task over story summaries from the recently released NarrativeQA Kočiskỳ et al. (2017)
dataset. It provides an interesting and a challenging test bed for reading comprehension as the summaries are long, and the answers to questions often do not occur in the summaries. We adopt the three-way attention modelWang (2018), an adapted version of the BiDAF Seo et al. (2016) span prediction model, in order to evaluate our method. We show that straightforward application of the answer selection model to entire summaries fails to outperform the model where the context is removed, demonstrating the weakness of current reading comprehension (RC) models . Inspired by Chen et al. (2017), we show that using TF-IDF to reduce context and applying global normalization on top of the reduced context, both significantly improve the performance. We observe that incorporating TF-IDF scores into the model with weighted global normalization helps improve performance more than either individually.
We view our contribution as twofold:
We introduce a novel weighted global normalization method for multiple choice question answering over long context documents.
We improve over the baseline of NarrativeQA answer selection task by a large margin, setting a competitive baseline for this interesting and challenging reading comprehension task.
2 Model Architecture
While span prediction models are not directly applicable to the answer selection task, the methods used to represent the context and question carry over. We base our model on the multiple choice architecture in Wang (2018). Taking inspiration from the popular BiDAF architecture Seo et al. (2016), the authors employ three-way attention over the context, question, and answer candidates to create and score answer representations. This section outlines our version of that architecture, denoted by T-Attn.
Word Embedding Layer.
We use pre-trained word embedding vectors to represent the tokens of query, context and all candidate answers.
Attention Flow Layer.
The interaction between query and context is modeled by computing a similarity matrix. This matrix is used to weigh query tokens to generate a query-aware representation for each context token. We compute query-aware-answer and context-aware-answer representations in a similar fashion. The representation of a token in terms of a sequence is computed as:
is ReLU activation.
We encode the query, context and the candidate answers by applying a Bi-GRU to a combination of the original and query/context-aware representations. Consequently, we obtain:-
where are token embeddings, and and are query-aware and context-aware token representations respectively.
The query and each candidate answers is re-weighted by weights learnt through a linear projection of the respective vectors. Context tokens are re-weighted by taking a bilinear attention with q.
is the hidden representation from the respective modeling layers. We adapt the output layer used inWang (2018) for multiple-choice answers by employing a feed-forward network () to compute scores for each candidate answer . The formulation is:-
Standard cross-entropy loss over all answer candidates is used for training.
In this section, we will first discuss the different methods, including our proposed method, used for handling the challenge of longer context documents. Since existing methods don’t apply to our task of answer selection as is, we also discuss the adjustments we made to these existing methods for comparing it with our proposed approach. For our task, we used the T-Attn model, described in Section 2, as our standard reading comprehension model.
3.1 Existing methods
The baseline is set by using the reading comprehension model, T-Attn model in our case, on the entire long context document.
Heuristic context reduction.
A simple method to make reading comprehension on longer contexts manageable is to use heuristic information retrieval methods, like TF-IDF as used by Chen et al. (2017), to reduce the context first and then apply any of the standard reading comprehension models to this reduced context.
We divide the summaries (the long context) into chunks comprising of approximate 40 tokens. These chunks are then ranked by their TF-IDF score with either the question (during validation and testing) or with the question and the gold answer (during training). We then apply the reading comprehension model to the top ranked chunks. We experiment with . Reducing the context in this way make it easier for the reading comprehension model to locate relevant information, but runs the risk of eliminating important context.
Clark and Gardner (2017) improve span prediction over multiple paragraphs by applying a reading comprehension model to each paragraph separately and globally normalizing span scores over all paragraphs.
This technique does not directly apply to the case of answer selection, as the correct answer is not a span and hence not tied to a specific paragraph. We implement an adjusted version of global normalization, in which we apply a reading comprehension model to each paragraph (in our case, each chunk
) separately, and sum all chunk scores over the answer candidates. The probability of answer candidatebeing correct, given chunks is
where is score given by chunk to answer candidate.
Normalizing in this manner encourages the model to produce lower scores for paragraphs that do not contain sufficient information to confidently answer the question.
3.2 Proposed method
Weighted global normalization.
The global normalization method relies on the reading comprehension model to learn what paragraphs contain useful information, rather than using the TF-IDF heuristic alone to eliminate paragraphs, which runs the risk of mistakenly culling useful context.
We incorporate the TF-IDF scores into reading comprehension model to re-weigh each chunk as follows:
where . These static scores for each chunk can be substituted with a learned function of these scores. For the purpose of demonstration, we use a simple MultiLayer Perceptron as the learning function, but it can be easily replaced with any function of choice. Hence, in Equation 12, , and we refer to this model as Wt-MLP.
The adapted Tri-Attention architecture with weighted global normalization for multiple choice answers, is shown in Figure 1.
|Weighted-Global-Norm (WGN-MLP)||Top 5 chunks||0.631||0.621|
|Weighted-Global-Norm (WGN)||Top 5 chunks||0.625||0.613|
|Global-Norm (GN)||Top 5 chunks||0.573||0.568|
|5 uniform chunks||0.545||0.531|
|Vanilla Tri-Attention (T-Attn)||Top 1 chunk||0.601||0.591|
|NarrativeQA Baseline (ASReader)||Full Summary||0.269||0.259|
The NarrativeQA dataset consists of 1572 movie scripts and books (Project Gutenberg) in the public domain, with 46765 corresponding questions. Each document is also accompanied by a corresponding summary extracted from Wikipedia. In this paper, we focus on the summaries as our dataset which have an average length of 659 tokens. For comparison, the average context size in the SQuAD dataset is less than 150 tokens. As described in Kočiskỳ et al. (2017), candidates for a question comprise of answers 111after removing duplicate answers to all the questions for a document. There are approximately 30 answer candidates per document.
4.2 Implementation Details
We split each summary into chunks of approximately 40 tokens, respecting sentence boundaries. We use 300 dimensional pre-trained Pennington et al. (2014) GloVe word embeddings, that are held fixed during training. We use a hidden dimension size of 128 in recurrent layers and 256 in linear layers. We use a 2-layer Bi-GRU in the modeling layer and a 3-layer feed-forward network for the output layer. The Wt-MLP
in weighted global normalization is 1 layer deep and takes following features as input:TF-IDF scores, max, min, average and standard deviation of answer scores for all chunks. A 0.2 dropout is used with Adam optimizer and a learning rate 0.002. The models converged within 10 epochs.
5 Results and Discussion
Table 1 reports results for our main experiments. We find that T-Attn without any context beats the NarrativeQA baseline by a large margin, suggesting the need for a stronger baseline. Surprisingly, T-Attn on full summaries performs no better than the No Context setting, implying that the model is unable to extract relevant information from a long context.
Providing the model with a reduced context of the top TF-IDF scored chunk (Top 1 chunk) leads to a significant gain (by +7). Global normalization also helps; uniformly sampling 5 chunks with global normalization (GN) yields a modest improvement over no-context, and taking the top 5 TF-IDF chunks rather than randomly sampling further improves performance, though not up to the level of the reduced context.
Both global normalization and TF-IDF scoring appear to provide a useful signal on the relevance of chunks. We found that combining the two in the form of weighted global normalization (WGN-MLP) outperforms both the globally normalized (by +6) and reduced context (by +3) experiments. The global normalization helps the model better tolerate the inclusion of likely, but not certainly, irrelevant chunks, while the weighting allows it to retain the strong signal from the TF-IDF scores.
Table 2 provides insight into the effect of global normalization. Global normalization models perform similarly to vanilla Tri-Attention trained on the top chunk when evaluated on the top chunk at test time, but degrade less in performance when evaluated on more chunks. Weighted global normalization in particular suffers only a minor penalty to performance from being evaluated on entire summaries. This effect may be even more pronounced on datasets where the top TF-IDF chunk is less reliable.
6 Conclusion and Future Work
This work introduces a method for improving answer selection on long documents through weighted global normalization of predictions over chunks of the documents. We show that applying our method to a span prediction model adapted for answer selection aids performance on long summaries in NarrativeQA and we strongly improve over the task baseline. In this work, we used a learned function of candidate and TF-IDF scores as the weights, but in principle the weighting function could take any form. For future work, we intend to explore the use of neural networks that takes into account the context and query to learn weights.
- Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051.
- Clark and Gardner (2017) Christopher Clark and Matt Gardner. 2017. Simple and effective multi-paragraph reading comprehension. CoRR, abs/1710.10723.
- Jia and Liang (2017) Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328.
- Kočiskỳ et al. (2017) Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2017. The narrativeqa reading comprehension challenge. arXiv preprint arXiv:1712.07040.
- Masala et al. (2017) Mihai Masala, Stefan Ruseti, and Traian Rebedea. 2017. Sentence selection with neural networks using string kernels. Procedia Computer Science, 112:1774–1782.
Pennington et al. (2014)
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.
Glove: Global vectors for word representation.
Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
- Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
- Seo et al. (2016) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603.
- Wang (2018) Liang Wang. 2018. Yuanfudao at semeval-2018 task 11: Three-way attention and relational knowledge for commonsense machine comprehension. CoRR, abs/1803.00191.
- Yu et al. (2018) Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541.