Question answering (QA) is a popular benchmark task in natural language understanding that aims to teach the machine to answer questions according to the given context. Earlier QA datasets such as SQuAD Rajpurkar et al. (2016a) mainly focus on evaluating the ability of finding answers with a single document per question. Subsequently, more challenging datasets that provide each question with multiple documents appear Joshi et al. (2017); Dunn et al. (2017). However, most questions of these datasets could still be solved using only one document. Recently, many datasets Welbl et al. (2018); Talmor and Berant (2018); Yang et al. (2018) that require multi-hop reasoning over multiple documents are constructed. In this paper, we focus on HotpotQA Yang et al. (2018), which is one of the representative multi-hop QA benchmarks. Each question in the HotpotQA Yang et al. (2018) is provided with a set of documents that includes both relevant and irrelevant documents. In addition to the exact answer spans, HotpotQA Yang et al. (2018) also offers supporting sentences that served as explanations to train and evaluate the QA systems.
Following the current mainstream pipeline for multi-hop QA Fang et al. (2020); Tu et al. (2020); Wu et al. (2021), our proposed framework also consists of a selector to minimize distracting information and a reader to provide exact answers and explanations. As for the design of the selector, most earlier works ignore the multi-hop nature of the task and treat the candidate documents or sentences independently. To address this problem, recent works include SAE Tu et al. (2020) and S2G Wu et al. (2021) apply a multi-head self-attention (MHSA) layer to encourage inter-document interactions. However, they still use a pre-trained language model (PLM) to encode each document separately before this MHSA layer, which cannot fully exploit the strong modeling and natural language understanding capacity of the PLMs and thus results in limited interactions. Notice that although S2G further adopts cascaded retrieval that fed with the text of the coarse-selected documents compared to SAE, these selectors aim to find all of the relevant documents at one time, ignoring the sequential nature of the relevant documents brought by the multi-hop reasoning.
Different from the above-mentioned selectors, our proposed selector consists of two stages, where one document is selected at a stage. Specifically, we first select the document that is most relevant to the question and then select the other one based on the question and the previously selected document. Figure 1 illustrates an example of this two-stage document selection process when we are answering multi-hop questions, which suggests that our two-stage selector shows great consistency to the reasoning process of humans. Concretely, we first arrive at document according to the question, which indicates that “Seven Brief Lessons on Physics is written by Carlo Rovelli”. Next, we identify the document containing “Carlo Rovelli has worked in France since 2000” using the question together with document . Our experiments demonstrate that with the help of the two-stage easy-to-hard manner, even simply adding a binary classification layer on a PLM could achieve strong performance.
After document selection, the filtered context is passed to the question answering module, namely the reader. Many existing works focus on guiding the readers to imitate the human reasoning process with elaborately designed pipelines, which can be roughly divided into two categories. One category is graph-based approaches, including SAE Tu et al. (2020) that treats sentences as graph nodes and HGN Fang et al. (2020)
that builds a hierarchical graph with nodes of different granularities. However, these methods require additional techniques, including named entity recognition and graph modeling, which inevitably lead to the growth of complexity and error accumulation. Due to these disadvantages, there have been works demonstrating that graph modeling may not be indispensableShao et al. (2020). Consequently, we also do not exploit any graph-based reasoning methods for the sake of simplicity. Another category is question decomposition methods, such as DecompRC Min et al. (2019) and ONUS Perez et al. (2020) that map one multi-hop, hard question to many single-hop, easy sub-questions. Although we appreciate the idea of decomposing the hard task into easy ones, the difficulty and complexity brought by the sub-question generation cannot be ignored.
Inspired by the question decomposition-based methods and the experience that children are usually learned from the very easy questions, we pose the question that whether it is too hard and confusing for models to directly learn to answer multi-hop questions from scratch. Therefore, our reader is firstly trained on a simpler single-hop dataset SQuAD Rajpurkar et al. (2016b) before performing the harder, multi-hop QA task. We assume that training on the single-hop dataset could initialize the model with better parameters and make it easier for the model to answer multi-hop questions. In addition, the process of learning from easy to hard is consistent with the widely accepted concept that there supposed to be a gradual progression of skill acquisition Liu et al. (2017); Roads et al. (2018); Song et al. (2019). Our experiments illustrate that the two-stage reader effectively boosts the performance, which verifies our assumption.
The main contributions of our work are summarized as follows:
We propose a simple yet effective multi-hop QA framework called FE2H, where both the document selector and reader are divided into two stages following the easy-to-hard manner.
We introduce a novel document selection module that iteratively performs binary classification tasks to select relevant documents by simply adding a prediction layer on a PLM.
The proposed reader is first trained on a single-hop QA dataset and then transferred into the multi-hop QA task, which is inspired by the progressive learning process of humans.
We comprehensively evaluate our model on the popular multi-hop QA benchmark HotpotQA. Experimental results show that the proposed FE2H can surpass the previous state-of-the-art by a large margin across all metrics.
2 Related Work
Selector for QA
Sentence or document selector aims to minimize noisy information, which has been proved effective for improving the performance and efficiency of the QA system. The study in Min et al. (2018) proposes a sentence selector for SQuAD Rajpurkar et al. (2016b) since questions can be answered within a single sentence. As for multi-hop QA, Wang et al. (2019) and Groeneveld et al. (2020) simply treat the documents and sentences separately, ignoring the interaction between them. Therefore, SAE Tu et al. (2020) and S2G Wu et al. (2021) append an MHSA layer to encourage document interaction. To acquire the finer-grained result, S2G Wu et al. (2021)
further adopt another cascaded paragraph retrieval module, which takes the text of the selected documents as input and subsequently explore deeper relationships between them. Since SAE and S2G attempt to locate all of the relevant documents at the same time, a naive binary classifier could not perform well. Therefore, SAETu et al. (2020) and S2G Wu et al. (2021) reformulate the classification objective into a ranking and scoring objective respectively to conform with the ranking nature of the selector. Different from the above selectors, we present a novel two-stage selector that neither selects the documents independently nor simultaneously.
Reader for multi-hop QA
Existing multi-hop QA readers can be roughly divided into two categories: reasoning based on intermediate results and reasoning based on graph. The first category can be further grouped by whether the intermediate result is represented in hidden state or free form text. As for the former group, QFE Nishida et al. (2019)
identifies relevant sentences by updating an RNN hidden state vector in each step, others update the representation vector of query and context iteratively to obtain the final answerDas et al. (2018); Feldman and El-Yaniv (2019). The representative works of the latter group mainly acquire the intermediate textual result by firstly decomposing the original question or generating a sub-question and then adopting off-the-shelf readers Qi et al. (2019); Min et al. (2019); Perez et al. (2020)
. The second category exploits graph neural networks to reason over the constructed graph. DFGNQiu et al. (2019) and CogQA Ding et al. (2019) focus on the entity graph while SAE Tu et al. (2020) treats sentences as graph nodes to identify supporting sentences. Furthermore, Tu et al. (2019), HGN Fang et al. (2020) and AMGN Li et al. (2021) reasoning over the heterogeneous graph with different types of nodes and edges. However, there are recent works that question whether the graph structure is necessary. Quark Groeneveld et al. (2020), C2F Reader Shao et al. (2020) and S2G Wu et al. (2021) demonstrate that the performances of the simple graph-free readers are on par with the graph-based reasoning methods. C2F Reader Shao et al. (2020) further claims that graph-attention can be considered as a special case of self-attention. In this paper, we propose a reader that simply follows a two-stage pipeline and free of any recurrent state updating and graph modeling.
3 Proposed Framework
We choose the distractor setting of HotpotQA Yang et al. (2018) as our testbed, where each question is equipped with 2 relevant documents and 8 distractors. A list of supporting sentences and an answer that could be an exact text span or “Yes/No” are also provided for evaluation. The overview of our proposed framework is shown in Figure 2. Following the traditional pipeline of multi-hop QA system, We first utilize a document selection module to construct almost noise-free context and then pass them to the downstream question answering module to extract supporting facts and answer span simultaneously.
3.1 Document Selection Module
At this phase, we aim to filter the distracting information and generate high-quality context, namely almost noise-free context, for the following question answering module. Note that although there are many works that perform recurrent reasoning using graph modeling and question decomposition when extracting supporting sentences and answers, existing selectors still extract relevant documents independently or simultaneously, which has been discussed in Section 1 and Section 2. Therefore, we assume that the recurrent reasoning manner should also be applied in a larger granularity, namely the document level. Recall that when we answer a multi-hop question, in most cases, we first select the document most relevant to the question and use this document together with the question to find another. Inspired by this insight, we divide the document selection into two steps, one document at a step, which is different from the existing methods.
The first stage
In this stage, we aim to select the document that is most relevant to the question. We feed the sequence “[CLS] + question + [SEP] + document + [SEP]” to a PLM ELECTRA Clark et al. (2020) and project the output of token “[CLS]” to calculate the relevant score for each candidate document and question . We label all the documents containing supporting facts as relevant and calculate the binary cross-entropy loss as:
where is the th question in the dataset, is the th candidate document of , is the label of the question-document pair , is the number of questions and is the size of each candidate document set, which is 10 for HotpotQA. We select the document with the highest relevance score and denote the chosen document as . That is, given question , .
The second stage
The aim of this stage is to find the other relevant document based on the previous result. Therefore, we generate the input as “[CLS] + question + [SEP] + + [SEP] + + [SEP]”, where is the document selected at the first step and is another candidate document. We calculate the relevance score of the candidate document as the previous step, which is denoted as . The binary cross-entropy loss is computed as:
where is the indicator function. Likewise, we select the document with the highest relevance score of this step and denote it as .
After the two-stage retrieval, the selected documents for each question are concatenated and passed to the following question answering module to serve as the high-quality context.
3.2 Question Answering Module
In this phase, we implement a multi-task model to extract the answers and the supporting facts of the multi-hop questions, which is illustrated in Figure 2. Firstly, we concatenate the context and the question as a sequence and use a PLM called ELECTRA Clark et al. (2020) to obtain the contextual representations of the input text. Subsequently, We exploit cross attention to enhance the interaction between the context and the question to acquire better context embeddings. Finally, a multi-task learning approach is adopted to jointly extract the answer and supporting sentences by simply adding a linear prediction layer.
Firstly, we are supposed to construct the input sequence as the input for ELECTRA Clark et al. (2020). Considering the “Yes/No” answer case, “yes” and “no” tokens are added to the beginning of the sequence. Besides, we add a “[UNK]” before each sentence in the context, whose contextual representation is used for supporting sentence classification. Therefore, our input sequence can be formulated as “[CLS] + yes + no + context + [SEP] + question + [SEP]”. We denote the ELECTRA output of the input as , the context output as and the question output as , where is the length of the input sequence, is the length of the context, is the length of the query, and is the hidden vector dimension of ELECTRA.
To improve the QA performance, we further add a cross attention interaction module to enhance the interaction between the context and query representations. At the same time, in order to obtain a better result and make the convergence faster during training, we use two cross attention blocks, including one with layer normalization and the other without layer normalization. The cross attention is computed as:
To extract the answer spans, we utilize a linear prediction layer on the context representations to identify the start and end position of the answer. The corresponding loss items are denoted as and respectively. As for the supporting facts prediction, we utilize a linear binary classifier on the output of the “[UNK]” that added before each sentence to decide whether a sentence is relevant or not. The classification objective of supporting facts is denoted as . Finally, we jointly optimize the above objectives as:
As discussed in Section 1, we assume that directly fine-tuned a PLM on a multi-hop QA task may be too hard and confusing for models. As a result, the model may answer the multi-hop questions with the single-hop shortcut, namely word-matching the question with a single sentence Jiang and Bansal (2019). Therefore, we first train our model on a single-hop QA dataset SQuAD Rajpurkar et al. (2016b) and then transfer the model to the multi-hop QA task. We suppose that the parameters that optimized by the single-hop QA task are much better than the random initialized parameters. Consequently, after learning single-hop QA task, the model tends to converge faster and achieve higher performance. Note that the model structure of the single-hop and multi-hop stage is the same.
|Baseline Model Yang et al. (2018)||45.60||59.02||20.32||64.49||10.83||40.16|
|QFE Nishida et al. (2019)||53.86||68.06||57.75||84.49||34.63||59.61|
|DFGN Qiu et al. (2019)||56.31||69.69||51.50||81.62||33.62||59.82|
|SAE-large Tu et al. (2020)||66.92||79.62||61.53||86.86||45.36||71.45|
|C2F Reader Shao et al. (2020)||67.98||81.24||60.81||87.63||44.67||72.73|
|HGN-large Fang et al. (2020)||69.22||82.19||62.76||88.47||47.11||74.21|
|AMGN+ Li et al. (2021)||70.53||83.37||63.57||88.83||47.77||75.24|
|S2G+EGA Wu et al. (2021)||70.92||83.44||63.86||88.68||48.76||75.47|
|FE2H on ELECTRA(ours)||69.54||82.69||64.78||88.71||48.46||74.90|
|FE2H on ALBERT(ours)||71.89||84.44||64.98||89.14||50.04||76.54|
We evaluate our model on the distractor setting of HotpotQA Yang et al. (2018). HotpotQA consists of 113k Wikipedia-based and crowdsourced question-answer pairs, including 90K training examples, 7.4K development examples and 7.4K test examples.
In the distractor setting, each question in the dataset is provided with a small set of documents that contains two supporting documents and eight irrelevant documents. HotpotQA Yang et al. (2018) also provides a list of supporting sentences for each question, which encourages model to explain the predictions. The Exact Match (EM) and F1 scores are used to evaluate models for both tasks and joint EM and F1 scores are used in the evaluation of the overall performance. The performance of our model on the test set is obtained by submitting our best model on the development set. Since the test set is not available, we report the performance of our comprehensive experiments on the development set.
4.2 Implementation Details
Our model is implemented based on the transformers of Hugging Face Wolf et al. (2020) and we adopt ELECTRA and ALBERT as our PLM to obtain the contextual embeddings of the questions and context. Our experiments are all performed on 2 or 3 NVIDIA A100 GPUs. The implementation details of our two-stage selector and reader are listed as follows.
We train two document selectors with a base and a large version of ELECTRA, respectively. Due to resource limitation, the ablations on document selection is only performed on ELECTRA-base. The number of epochs is set to 3 and batch size is set to 12. We use BERT-Adam with learning rate of 2e-5 for the optimization. The hyper-parameters of our two selectors of different stages are same while the model parameters are different and trained sequentially.
As for the question answering phase, we train the FE2H on ELECTRA model with batch size of 16 with 2 NVIDIA A100 GPUs, learning rate of 1.5e-5, warm-up rate of 0.1, and L2 weight decay of 0.01. Besides, we use ELECTRA-large as the PLM of this phase. It takes about 2 hours per epoch and 6.5 hours in total to train a reader of a single stage. we train the FE2H on ALBERT model with batch size of 12 with 3 NVIDIA A100 GPUs, learning rate of 1.0e-5, warm-up rate of 0.1, and L2 weight decay of 0.01. Besides, we use Albert-xxlarge-v2 as the PLM of this phase. It takes about 25 hours in total to train a reader of a single stage.
4.3 Experimental Results
Results on HotpotQA
Table 1 shows the performance of published advanced methods on the test set of HotpotQA in the distractor setting. Despite the simplicity of our model, Our FE2H on ALBERT is the state-of-the-art model on the leaderboard111Leaderboard available at https://hotpotqa.github.io. Our model achieves the best result in all metrics, demonstrating the strong multi-hop reasoning ability of our model. The Joint EM improvement is 1.28% and the Joint F1 improvement is 1.07% among all the published methods. And the Joint F1 improvement is 0.82% among all the submitted methods including SAE+1.
We compare our selector with the ones of three published advanced works, that is, HGN, SAE and S2G, which also apply carefully designed selectors to retrieve relevant documents for the downstream readers. We use EM and F1 to evaluate the performance of the document selectors. As shown in Table 2, our selector outperforms these three strong baselines. Even the base version of our selector achieves competitive performance compared to the previous best selector.
|1st Stage + Top 2||89.10||94.41|
|1st Stage + THOLD||85.10||87.01|
|1st Stage (AW) + Top 2||84.17||91.78|
|1st Stage (AW) + 2nd Stage||92.72||96.09|
Two-stage Selector Ablation
The results of the two-stage selector ablation are shown in Table 3. To prove the validity of out two-stage selector, we remove the second stage and evaluate the results. Results illustrate that, given the output of the first stage, no matter we select the top 2 documents or select the documents whose scores above a certain threshold, both EM and F1 score drop significantly. Note that both SAE and S2G believe that the documents containing the answer span (i.e., gold documents) are more important for downstream tasks and thus assign these documents with higher score. However, we assume that relevant documents are of equal importance, especially for the first stage of our document selection. Since we do not reformulate the binary classification objective to ranking-related objectives like SAE and S2G, we assign greater weights to the gold documents to verify this assumption. Results illustrate that performance drops significantly after reassigning the weights of the relevant documents as above at the first stage. It is worth pointing that the performance of single-stage setting degrades more compared to the two-stage setting, indicating that directly guiding the selector to identify the gold documents is infeasible, thus both relevant documents are equally important.
|Setting||Joint F1||Joint EM|
|- Cross Attention w/ LN||75.14||48.37|
|- Cross Attention w/o LN||75.07||48.37|
|- Cross Attention Block||75.20||48.19|
|Two Cross Attention w/ LN||74.98||48.31|
|Two Cross Attention w/o LN||75.18||48.29|
QA Model Ablation
Due to the computing resource limitations, we only use a single-stage reader, i.e. the reader without training on the single-hop QA dataset, to perform the ablation of QA model structure, which is shown in Table 4. Specifically, we investigate the effect of different interaction layers between the question and the context on the performance of extracting supporting facts and answers.
Recall that our QA model exploits a cross attention block that consists of a cross attention layer with layer normalization and one without. We first ablate the cross attention block of our QA model. As shown in Table 4, both the joint F1 score and joint EM drops whether one or two cross attention layers are removed. Furthermore, in order to explore the effect of the asymmetric structure of our cross attention block, we replace it with two same cross attention layers with and without layer normalization, respectively. Experimental results show that our cross attention block outperforms the above two settings. We hypothesis that this is because the one with layer normalization tends to converge faster Ba et al. (2016) and the one without layer normalization tends to explore a better global solution due to the randomness of the layer. As a result, our cross attention block combines the advantages of both and achieve better performance.
In addition to the cross attention layer, we attempt to explore the effect of other layers on enhancing the interaction between the question and the context, which includes self-attention, co-attention, bi-attention, and feed-forward layers. These layers are directly added to the PLM during experiments. Note that our QA model without the cross attention block is equivalent to directly predicting results based on the output of PLM, namely the baseline QA model, while our experiments show that adding these layers all lead to performance degradation compared to the baseline. For these degradations, our intuitive explanations are listed as follows. Since the feed-forward network only takes the context embeddings as input, thus prohibits the interaction between the question and context, which probably results in performance degradation. As for self-attention, recall that the PLM has already contained many self-attention layers, thus learning an extra self-attention layer from scratch may not help. Moreover, because the interactions modeled by bi-attentionSeo et al. (2017) and co-attention Xiong et al. (2017) are more complicated compared to the ones of self-attention and cross-attention, we suppose that it is hard to train a model with many pre-trained self-attention layers and a random-initialized bi-attention or co-attention layer.
In summary, however, we come to the conclusion that though our cross attention block improves the performance, the improvement is not significant and many other interaction mechanisms even negatively affect the performance. That is to say, without any additional computations, directly fine-tuned PLMs could achieve competitive performance on the downstream tasks.
|PLM||First Stage||Second Stage||Joint F1||Joint EM|
Two-stage Reader Ablation
To demonstrate the effectiveness of the two-stage manner of our reader, we perform our ablation studies using a smaller PLM BERT-base Devlin et al. (2019) and a larger PLM ELECTRA-large Clark et al. (2020). The results are illustrated in Table 5, which demonstrate that the two-stage manner brings more significant performance improvement on BERT Devlin et al. (2019) than ELECTRA. Specifically, joint F1 drops 1.45% and joint EM drops 2.04% when only training on the multi-hop QA dataset. We hypothesis that this is because larger PLM like ELECTRA has learned much more common knowledge and obtained stronger natural language understanding ability, thus being further fine-tuned on simple tasks like single-hop QA may be of limited help. While as for smaller PLM like BERT Devlin et al. (2019), single-hop QA datasets could offer more task-specific knowledge to the PLM and effectively bridge the gap between the PLM and the multi-hop QA task. Therefore, we recommend the two-stage reader, especially under the circumstances that resources are limited and only small PLMs are permitted.
Subsequently, we perform experiments using both single-hop and multi-hop QA datasets but with different training strategies. As illustrated in Table 5, performance drops when we perform single-hop and multi-hop QA tasks simultaneously, which indicates that the performance improvement of our two-stage reader is not only brought by the use of more labeled QA pairs, but also brought by the easy-to-hard two-stage manner. Notice that only using the single-hop dataset at the first stage and using both datasets at the second stage also leads to performance degradation. We presume that this is because once the model has learned the easy task, feeding it with the datasets of both tasks in the next step is of little use and even probably negatively impact the training results.
We propose FE2H, a simple yet effective framework for multi-hop QA that divides both the document selection and question answering into two stages following an easy-to-hard manner. Experimental results demonstrate that since we cannot feed all of the candidate documents to the PLMs at a time due to the input length limitation, taking the multi-hop reasoning nature into consideration at the document selection phase significantly improves the overall performance. As for the subsequent QA phase, thanks to the great natural language understanding ability of the PLMs, the performance of our simple two-stage reader is better than the state-of-the-art approaches without any graph structure and explicit reasoning chains. We hope this work could facilitate more simple yet powerful multi-hop QA approaches with the help of the advanced PLMs.
We thank many colleagues at Nanjing University for their help, particularly Qing-Long Zhang, for useful discussion.
- Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv:1607.06450.
- Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In ICLR.
- Das et al. (2018) Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2018. Multi-step retriever-reader interaction for scalable open-domain question answering. In ICLR.
- Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
- Ding et al. (2019) Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. 2019. Cognitive graph for multi-hop reading comprehension at scale. In ACL.
- Dunn et al. (2017) Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. CoRR, abs/1704.05179.
- Fang et al. (2020) Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. 2020. Hierarchical graph network for multi-hop question answering. In EMNLP.
- Feldman and El-Yaniv (2019) Yair Feldman and Ran El-Yaniv. 2019. Multi-hop paragraph retrieval for open-domain question answering. In ACL.
- Groeneveld et al. (2020) Dirk Groeneveld, Tushar Khot, Mausam, and Ashish Sabharwal. 2020. A simple yet strong pipeline for HotpotQA. In EMNLP.
- Jiang and Bansal (2019) Yichen Jiang and Mohit Bansal. 2019. Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop QA. In ACL.
- Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL.
- Li et al. (2021) Ronghan Li, Lifang Wang, Shengli Wang, and Zejun Jiang. 2021. Asynchronous multi-grained graph network for interpretable multi-hop reading comprehension. In IJCAI.
Liu et al. (2017)
Weiwei Liu, Ivor W Tsang, and Klaus-Robert Müller. 2017.
An easy-to-hard learning paradigm for multiple classes and multiple
The Journal of Machine Learning Research, 18.
- Min et al. (2018) Sewon Min, Victor Zhong, Richard Socher, and Caiming Xiong. 2018. Efficient and robust question answering from minimal context over documents. In ACL.
- Min et al. (2019) Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019. Multi-hop reading comprehension through question decomposition and rescoring. In ACL.
- Nishida et al. (2019) Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito, Hisako Asano, and Junji Tomita. 2019. Answering while summarizing: Multi-task learning for multi-hop QA with evidence extraction. In ACL.
- Perez et al. (2020) Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. Unsupervised question decomposition for question answering. In EMNLP.
- Qi et al. (2019) Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D. Manning. 2019. Answering complex open-domain questions through iterative query generation. In EMNLP-IJCNLP.
- Qiu et al. (2019) Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. 2019. Dynamically fused graph network for multi-hop reasoning. In ACL.
Rajpurkar et al. (2016a)
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.
questions for machine comprehension of text.
Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
- Rajpurkar et al. (2016b) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016b. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP.
- Roads et al. (2018) Brett D Roads, Buyun Xu, June K Robinson, and James W Tanaka. 2018. The easy-to-hard training advantage with real-world medical images. Cognitive Research: Principles and Implications, 3(1):1–13.
- Seo et al. (2017) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. ICLR.
- Shao et al. (2020) Nan Shao, Yiming Cui, Ting Liu, Shijin Wang, and Guoping Hu. 2020. Is Graph Structure Necessary for Multi-hop Question Answering? In EMNLP.
- Song et al. (2019) Shaoyue Song, Hongkai Yu, Zhenjiang Miao, Dazhou Guo, Wei Ke, Cong Ma, and Song Wang. 2019. An easy-to-hard learning strategy for within-image co-saliency detection. Neurocomputing, 358:166–176.
- Talmor and Berant (2018) Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In NAACL-HLT.
- Tu et al. (2019) Ming Tu, Guangtao Wang, Jing Huang, Yun Tang, Xiaodong He, and Bowen Zhou. 2019. Multi-hop reading comprehension across multiple documents by reasoning over heterogeneous graphs. In ACL.
- Tu et al. (2020) Ming Tu, Bowen Zhou, et al. 2020. Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents. In AAAI.
- Wang et al. (2019) Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019. Multi-passage BERT: A globally normalized BERT model for open-domain question answering. In EMNLP-IJCNLP.
- Welbl et al. (2018) Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. TACL, 6.
- Wolf et al. (2020) Thomas Wolf, Alexander M. Rush, et al. 2020. Transformers: State-of-the-art natural language processing. In EMNLP.
- Wu et al. (2021) Bohong Wu, Zhuosheng Zhang, and Hai Zhao. 2021. Graph-free Multi-hop Reading Comprehension: A Select-to-Guide Strategy. arXiv:2107.11823.
- Xiong et al. (2017) Caiming Xiong, Victor Zhong, and Richard Socher. 2017. Dynamic coattention networks for question answering. ICLR.
- Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP.