Generating Question-Answer Hierarchies

The process of knowledge acquisition can be viewed as a question-answer game between a student and a teacher in which the student typically starts by asking broad, open-ended questions before drilling down into specifics (Hintikka, 1981; Hakkarainen and Sintonen, 2002). This pedagogical perspective motivates a new way of representing documents. In this paper, we present SQUASH (Specificity-controlled Question-Answer Hierarchies), a novel and challenging text generation task that converts an input document into a hierarchy of question-answer pairs. Users can click on high-level questions (e.g., "Why did Frodo leave the Fellowship?") to reveal related but more specific questions (e.g., "Who did Frodo leave with?"). Using a question taxonomy loosely based on Lehnert (1978), we classify questions in existing reading comprehension datasets as either "general" or "specific". We then use these labels as input to a pipelined system centered around a conditional neural language model. We extensively evaluate the quality of the generated QA hierarchies through crowdsourced experiments and report strong empirical results.


page 7

page 14


OneStop QAMaker: Extract Question-Answer Pairs from Text in a One-Stop Approach

Large-scale question-answer (QA) pairs are critical for advancing resear...

Know What You Don't Know: Unanswerable Questions for SQuAD

Extractive reading comprehension systems can often locate the correct an...

AnswerQuest: A System for Generating Question-Answer Items from Multi-Paragraph Documents

One strategy for facilitating reading comprehension is to present inform...

BERT-based distractor generation for Swedish reading comprehension questions using a small-scale dataset

An important part when constructing multiple-choice questions (MCQs) for...

Variational Question-Answer Pair Generation for Machine Reading Comprehension

We present a deep generative model of question-answer (QA) pairs for mac...

O.D.E.S. : An Online Dynamic Examination System based on a CMS Wordpress plugin

This paper describes the online dynamic examination application plugin n...

I Do Not Understand What I Cannot Define: Automatic Question Generation With Pedagogically-Driven Content Selection

Most learners fail to develop deep text comprehension when reading textb...

1 Introduction

Q: What is this paper about?
A: We present a novel text generation task which converts an input document into a model-generated hierarchy of question-answer (QA) pairs arranged in a top-down tree structure (Figure 1). Questions at higher levels of the tree are broad and open-ended while questions at lower levels ask about more specific factoids. An entire document has multiple root nodes (“key ideas”) that unfold into a forest of question trees. While readers are initially shown only the root nodes of the question trees, they can “browse” the document by clicking on root nodes of interest to reveal more fine-grained related information. We call our task SQUASH  (Specificity-controlled Question Answer Hierarchies).

Figure 1: A subset of the QA hierarchy generated by our SQUASH  system that consists of general and specific questions with extractive answers.

Q: Why represent a document with QA pairs?111Our introduction is itself an example of the QA format. Other academic papers such as  Henderson2018DeepRL have also used this format to effectively present information.
A:  Questions and answers (QA) play a critical role in scientific inquiry, information-seeking dialogue and knowledge acquisition Hintikka (1981, 1988); Stede and Schlangen (2004). For example, web users often use QA pairs to manage and share knowledge (Wagner, 2004; Wagner and Bolloju, 2005; Gruber, 2008). Additionally, unstructured lists of “frequently asked questions” (FAQs) are regularly deployed at scale to present information. Industry studies have demonstrated their effectiveness at cutting costs associated with answering customer calls or hiring technical experts Davenport et al. (1998). Automating the generation of QA pairs can thus be of immense value to companies and web communities.

Q: Why add hierarchical structure to QA pairs?

  While unstructured FAQs are useful, pedagogical applications benefit from additional hierarchical organization. hakkarainen2002interrogative show that students learn concepts effectively by first asking general, explanation-seeking questions before drilling down into more specific questions. More generally, hierarchies break up content into smaller, more digestable chunks. User studies demonstrate a strong preference for hierarchies in document summarization 

Buyukkokten et al. (2001); Christensen et al. (2014) since they help readers easily identify and explore key topics Zhang et al. (2017).

Q: How do we build systems for SQUASH ?
A: We leverage the abundance of reading comprehension QA datasets to train a pipelined system for SQUASH . One major challenge is the lack of labeled hierarchical structure within existing QA datasets; we tackle this issue in Section 2 by using the question taxonomy of lehnert1978process to classify questions in these datasets as either general or specific. We then condition a neural question generation system on these two classes, which enables us to generate both types of questions from a paragraph. We filter and structure these outputs using the techniques described in Section 3.

Q: How do we evaluate our SQUASH  pipeline?
A:  Our crowdsourced evaluation (Section 4) focuses on fundamental properties of the generated output such as QA quality, relevance, and hierarchical correctness. Our work is a first step towards integrating QA generation into document understanding; as such, we do not directly evaluate how useful SQUASH  output is for downstream pedagogical applications. Instead, a detailed qualitative analysis (Section 5) identifies challenges that need to be addressed before SQUASH  can be deployed to real users.

Q: What are our main contributions?
A1: A method to classify questions according to their specificity based on  lehnert1978process.
A2: A model controlling specificity of generated questions, unlike prior work on QA generation.
A3: A novel text generation task ( SQUASH ), which converts documents into specificity-based hierarchies of QA pairs.
A4: A pipelined system to tackle SQUASH  along with crowdsourced methods to evaluate it.

Q: How can the community build on this work?
A:  We have released our codebase, crowdsourcing templates for evaluation, and a live demonstration of our system at Additionally, we outline guidelines for future work in Section 7.

2 Obtaining training data for SQUASH

The proliferation of reading comprehension datasets like SQuAD Rajpurkar et al. (2016, 2018) has enabled state-of-the-art neural question generation systems Du et al. (2017); Kim et al. (2018). However, these systems are trained for individual question generation, while the goal of SQUASH  is to produce a general-to-specific hierarchy of QA pairs. Recently-released conversational QA datasets like QuAC Choi et al. (2018) and CoQA Reddy et al. (2018) contain a sequential arrangement of QA pairs, but question specificity is not explicitly marked.222“Teachers” in the QuAC set-up can encourage “students” to ask a follow-up question, but we cannot use these annotations to infer a hierarchy because students are not required to actually follow their teachers’ directions. Motivated by the lack of hierarchical QA datasets, we automatically classify questions in SQuAD, QuAC and CoQA according to their specificity using a combination of rule-based and automatic approaches.

Conceptual class Specificity Question asks for… Sample templates
Causal Antecendent, Goal Oriented, Enablement, Causal Consequent, Expectational general the reason for occurrence of an event and the consequences of it Why …, What happened after / before …, What was the cause / reason / purpose …, What led to …
general a procedure / mechanism How question with VERB parent for How in dependency tree
Judgemental general a listener’s opinion Words like you, your present
Concept Completion, Feature Specification general  or
fill-in-the-blank information Where / When / Who …
(“specific” templates)
Quantification specific an amount How many / long …
Verification, Disjunctive yes-no Yes-No answers first word is VERB
Request N/A an act to be performed (absent in datasets)
Table 1: The 13 conceptual categories of lehnert1978process and some templates to identify them and their specificity.

2.1 Rules for specificity classification

What makes one question more specific than another? Our scheme for classifying question specificity maps each of the 13 conceptual question categories defined by lehnert1978process to three coarser labels: general, specific, or yes-no.333We add a third category for yes-no questions as they are difficult to classify as either general or specific. As a result of this mapping, specific questions usually ask for low-level information (e.g., entities or numerics), while general questions ask for broader overviews (e.g., “what happened in 1999?”) or causal information (e.g, “why did…”). Many question categories can be reliably identified using simple templates and rules; A complete list is provided in Table 1.444Questions in lehnert1978process were classified using a conceptual dependency parser Schank (1972). We could not find a modern implementation of this parser and thus decided to use a rule-based approach that relies on spaCy 2.0 (Honnibal and Montani, 2017) for all preprocessing.

Classifying questions not covered by templates: If a question does not satisfy any template or rule, how do we assign it a label? We manage to classify roughly half of all questions with our templates and rules (Table A1); for the remaining half, we resort to a data-driven approach. First, we manually label 1000 questions in QuAC555We use QuAC because its design encourages a higher percentage of general questions than other datasets, as the question-asker was unable to read the document to formulate more specific questions. using our specificity labels. This annotated data is then fed to a single-layer CNN binary classifier Kim (2014) using ELMo contextualized embeddings Peters et al. (2018).666Implemented in AllenNLP Gardner et al. (2018). On a 85%-15% train-validation split, we achieve a high classification accuracy of 91%. The classifier also transfers to other datasets: on 100 manually labeled CoQA questions, we achieve a classification accuracy of 80%. To obtain our final dataset (Table 2), we run our rule-based approach on all questions in SQuAD 2.0, QuAC, and CoQA and apply our classifier to label questions that were not covered by the rules. We further evaluate the specificity of the questions generated by our final system using a crowdsourced study in Section 4.3.

Dataset Size general specific yes-no
SQuAD 86.8k 28.2% 69.7% 2.1%
QuAC 65.2k 34.9% 33.5% 31.6%
CoQA 105.6k 23.6% 54.9% 21.5%
All 257.6k 28.0% 54.5% 17.5%
Table 2: Distribution of classes in the final datasets. We add some analysis on this distribution in Appendix A.
Figure 2: An overview of the process by which we generate a pair of general-specific questions, which consists of feeding input data (“RC” is Reading Comprehension) through various modules, including a question classifier and a multi-stage pipeline for question generation, answering, and filtering.

3 A pipeline for SQUASH ing documents

To SQUASH  documents, we build a pipelined system (Figure 2) that takes a single paragraph as input and produces a hierarchy of QA pairs as output; for multi-paragraph documents, we SQUASH  each paragraph independently of the rest. At a high level, the pipeline consists of five steps: (1) answer span selection, (2) question generation conditioned on answer spans and specificity labels, (3) extractively answering generated questions, (4) filtering out bad QA pairs, and (5) structuring the remaining pairs into a general-to-specific hierarchy. The remainder of this section describes each step in more detail.

3.1 Answer span selection

Our pipeline begins by selecting an answer span from which to generate a question. To train the system, we can use ground-truth answer spans from our labeled datasets, but at test time how do we select answer spans? Our solution is to consider all individual sentences in the input paragraph as potential answer spans (to generate general and specific questions), along with all entities and numerics (for just specific  questions). We did not use data-driven sequence tagging approaches like previous work (Du and Cardie, 2017, 2018), since our preliminary experiments with such approaches yielded poor results on QuAC.777We hypothesize that answer span identification on QuAC is difficult because the task design encouraged “teachers” to provide more information than just the minimal answer span. More details are provided in Appendix C.

3.2 Conditional question generation

Given a paragraph, answer span, and desired specificity label, we train a neural encoder-decoder model on all three reading comprehension datasets (SQuAD, QuAC and CoQA) to generate an appropriate question.

Data preprocessing: At training time, we use the ground-truth answer spans from these datasets as input to the question generator. To improve the quality of specific questions generated from sentence spans, we use the extractive evidence spans for CoQA instances Reddy et al. (2018) instead of the shorter, partially abstractive answer spans Yatskar (2019). In all datasets, we remove unanswerable questions and questions whose answers span multiple paragraphs. A few very generic questions (e.g. “what happened in this article?”) were manually identified removed from the training dataset. Some other questions (e.g., “where was he born?”) are duplicated many times in the dataset; we downsample such questions to a maximum limit of 10. Finally, we preprocess both paragraphs and questions using byte-pair encoding Sennrich et al. (2016).

Architecture details: We use a two-layer biLSTM encoder and a single-layer LSTM (Hochreiter and Schmidhuber, 1997) decoder with soft attention (Bahdanau et al., 2015) to generate questions, similar to Du et al. (2017). Our architecture is augmented with a copy mechanism See et al. (2017) over the encoded paragraph representations. Answer spans are marked with <SOA> and <EOA> tokens in the paragraph, and representations for tokens within the answer span are attended to by a separate attention head. We condition the decoder on the specificity class (general, specific and yes-no)888While we do not use yes-no questions at test time, we keep this class to avoid losing a significant proportion of training data.

by concatenating an embedding for the ground-truth class to the input of each time step. We implement models in PyTorch

v0.4 Paszke et al. (2017)

, and the best-performing model achieves a perplexity of 11.1 on the validation set. Other hyperparameters details are provided in

Appendix B.

Test time usage: At test time, the question generation module is supplied with answer spans and class labels as described in Section 3.1. To promote diversity, we over-generate prospective candidates Heilman and Smith (2010)

for every answer span and later prune them. Specifically, we use beam search with a beam size of 3 to generate three highly-probable question candidates. As these candidates are often generic, we additionally use

top-k random sampling Fan et al. (2018) with , a recently-proposed diversity-promoting decoding algorithm, to generate ten more question candidates per answer span. Hence, for every answer span we generate 13 question candidates. We discuss issues with using just standard beam search for question generation in Section 5.1.

3.3 Answering generated questions

While we condition our question generation model on pre-selected answer spans, the generated questions may not always correspond to these input spans. Sometimes, the generated questions are either unanswerable or answered by a different span in the paragraph. By running a pretrained QA model over the generated questions, we can detect questions whose answers do not match their original input spans and filter them out. The predicted answer for many questions has partial overlap with the original answer span; in these cases, we display the predicted answer span during evaluation, as a qualitative inspection shows that the predicted answer is more often closer to the correct answer. For all of our experiments, we use the AllenNLP implementation of the BiDAF++ question answering model of ChoiQuAC2018 trained on QuAC with no dialog context.

3.4 Question filtering

After over-generating candidate questions from a single answer span, we use simple heuristics to filter out low-quality QA pairs. We remove generic and duplicate question candidates

999Running Top-k random sampling multiple times can produce duplicate candidates, including those already in the top beams. and pass the remaining QA pairs through the multi-stage question filtering process described below.

Irrelevant or repeated entities: Top-k random sampling often generates irrelevant questions; we reduce their incidence by removing any candidates that contain nouns or entities unspecified in the passage. As with other neural text generation systems Holtzman et al. (2018), we commonly observe repetition in the generated questions and deal with this phenomenon by removing candidates with repeated nouns or entities.

Unanswerable or low answer overlap:

We remove all candidates marked as “unanswerable” by the question answering model, which prunes 39.3% of non-duplicate question candidates. These candidates are generally grammatically correct but considered irrelevant to the original paragraph by the question answering model. Next, we compute the overlap between original and predicted answer span by computing word-level precision and recall 

Rajpurkar et al. (2016). For general questions generated from sentence spans, we attempt to maximize recall by setting a minimum recall threshold of 0.3.101010Minimum thresholds were qualitatively chosen based on the specificity type. Similarly, we maximize recall for specific questions generated from named entities with a minimum recall constraint of 0.8. Finally, for specific questions generated from sentence spans, we set a minimum precision threshold of 1.0, which filters out questions whose answers are not completely present in the ground-truth sentence.

Low generation probability: If multiple candidates remain after applying the above filtering criteria, we select the most probable candidate for each answer span. specific questions generated from sentences are an exception to this rule: for these questions, we select the ten most probable candidates, as there might be multiple question-worthy bits of information in a single sentence. If no candidates remain, in some cases111111For example, if no valid general questions for the entire paragraph are generated. we use a fallback mechanism that sequentially ignores filters to retain more candidates.

3.5 Forming a QA hierarchy

The output of the filtering module is an unstructured list of general and specific QA pairs generated from a single paragraph. Figure 3 shows how we group these questions into a meaningful hierarchy. First, we choose a parent for each specific question by maximizing the overlap (word-level precision) of its predicted answer with the predicted answer for every general question. If a specific question’s answer does not overlap with any general question’s answer (e.g., “Dagobah” and “destroy the Sith”) we map it to the closest general question whose answer occurs before the specific question’s answer (“What happened in the battle …?”).121212This heuristic is justified because users read general questions before specific ones in our interface.

Figure 3: Procedure used to form a QA hierarchy. The predicted answers for GQs (general questions), are underlined in blue. The predicted answers for SQs (specific questions) are highlighted in red.

4 Evaluation

Experiment Generated Gold
Score Fleiss  Score Fleiss 
Is this question well-formed? 85.8% 0.65 93.3% 0.54
Is this question relevant? 78.7% 0.36 83.3% 0.41
    (among well-formed) 81.1% 0.39 83.3% 0.40
Does the span partially contain the answer? 85.3% 0.45 81.1% 0.43
    (among well-formed) 87.6% 0.48 82.1% 0.42
    (among well-formed and relevant) 94.9% 0.41 92.9% 0.44
Does the span completely contain the answer? 74.1% 0.36 70.0% 0.37
    (among well-formed) 76.9% 0.36 70.2% 0.39
    (among well-formed and relevant) 85.4% 0.30 80.0% 0.42
Table 3: Human evaluations demonstrate the high individual QA quality of our pipeline’s outputs. All interannotator agreement scores (Fleiss ) show “fair” to “substantial” agreement Landis and Koch (1977).

We evaluate our SQUASH  pipeline on documents from the QuAC development set using a variety of crowdsourced131313All our crowdsourced experiments were conducted on the Figure Eight platform with three annotators per example (scores calculated by counting examples with two or more correct judgments). We hired annotators from predominantly English-speaking countries with a rating of at least Level 2, and we paid them between 3 and 4 cents per judgment. experiments. Concretely, we evaluate the quality and relevance of individual questions, the relationship between generated questions and predicted answers, and the structural properties of the QA hierarchy. We emphasize that our experiments examine only the quality of a SQUASH ed document, not its actual usefulness to downstream users. Evaluating usefulness (e.g., measuring if SQUASH  is more helpful than the input document) requires systematic and targeted human studies (Buyukkokten et al., 2001) that are beyond the scope of this work.

4.1 Individual question quality and relevance

Our first evaluation measures whether questions generated by our system are well-formed (i.e., grammatical and pragmatic). We ask crowd workers whether or not a given question is both grammatical and meaningful.141414As “meaningful” is potentially a confusing term for crowd workers, we ran another experiment asking only for grammatical correctness and achieved very similar results. For this evaluation, we acquire judgments for 200 generated QA pairs and 100 gold QA pairs151515Results on this experiment were computed after removing 3 duplicate generated questions and 10 duplicate gold questions. from the QuAC validation set (with an equal split between general and specific questions). The first row of Table 3 shows that 85.8% of generated questions satisfy this criterion with a high agreement across workers.

Question relevance: How many generated questions are actually relevant to the input paragraph? While the percentage of unanswerable questions that were generated offers some insight into this question, we removed all of them during the filtering pipeline (Section 3.4). Hence, we display an input paragraph and generated question to crowd workers (using the same data as the previous well-formedness evaluation) and ask whether or not the paragraph contains the answer to the question. The second row of Table 3 shows that 78.7% of our questions are relevant to the paragraph, compared to 83.3% of gold questions.

4.2 Individual answer validity

Is the predicted answer actually a valid answer to the generated question? In our filtering process, we automatically measured answer overlap between the input answer span and the predicted answer span and used the results to remove low-overlap QA pairs. To evaluate answer recall after filtering, we perform a crowdsourced evaluation on the same 300 QA pairs as above by asking crowdworkers whether or not a predicted answer span contains the answer to the question. We also experiment with a more relaxed variant (partially contains instead of completely contains) and report results for both task designs in the third and fourth rows of Table 3. Over 85% of predicted spans partially contain the answer to the generated question, and this number increases if we consider only questions that were previously labeled as well-formed and relevant. The lower gold performance is due to the contextual nature of the gold QA pairs in QuAC, which causes some questions to be meaningless in isolation (e.g.“What did she do next?” has unresolvable coreferences).

Experiment Score Fleiss 
Which question type asks for more information? 89.5% 0.57
Which specific question is closer to general QA?
        different paragraph 77.0% 0.47
        same paragraph 64.0% 0.30
Table 4: Human evaluation of the structural correctness of our system. The labels “different / same paragraph” refer to the location of the intruder question. The results show the accuracy of specificity and hierarchies.
Figure 4: SQUASH  question hierarchies generated by our system with reference snippets. Questions in the hierarchy are of the correct specificity class (i.e., general, specific).

4.3 Structural correctness

To examine the hierachical structure of SQUASH ed documents, we conduct three experiments.

How faithful are output questions to input specificity? First, we investigate whether our model is actually generating questions with the correct specificity label. We run our specificity classifier (Section 2) over 400 randomly sampled questions (50% general, 50% specific) and obtain a high classification accuracy of 91%.161616Accuracy computed after removing 19 duplicates. This automatic evaluation suggests the model is capable of generating different types of questions.

Are general questions more representative of a paragraph than specific questions? To see if general questions really do provide more high-level information, we sample 200 general-specific question pairs171717We avoid gold-standard control experiments for structural correctness tests since questions in the QuAC dataset were not generated with a hierarchical structure in mind. Pilot studies using our question grouping module on gold data led to sparse hierarchical structures which were not favored by our crowd workers. grouped together as described in Section 3.5. For each pair of questions (without showing answers), we ask crowd workers to choose the question which, if answered, would give them more information about the paragraph. As shown in Table 4, in 89.5% instances the general question is preferred over the specific one, which confirms the strength of our specificity-controlled question generation system.181818We also ran a pilot study asking workers “Which question has a longer answer?” and observed a higher preference of 98.6% for general questions.

How related are specific questions to their parent general question? Finally, we investigate the effectiveness of our question grouping strategy, which bins multiple specific QA pairs under a single general QA pair. We show crowd workers a reference general QA pair and ask them to choose the most related specific  question given two choices, one of which is the system’s output and the other an intruder question. We randomly select intruder specific questions from either a different paragraph within the same document or a different group within the same paragraph. As shown in Table 4, crowd workers prefer the system’s generated specific question with higher than random chance (50%) regardless of where the intruder comes from. As expected, the preference and agreement is higher when intruder questions come from different paragraphs, since groups within the same paragraph often contain related information (Section 5.2).

Figure 5: Two SQUASH  outputs generated by our system. The William Bryan example has interesting general questions. The Paul Weston example showcases several mistakes our model makes.

5 Qualitative Analysis

In this section we analyze outputs (Figure 4, Figure 5) of our pipeline and identify its strengths and weaknesses. We additionally provide more examples in the appendix (Figure A1).

5.1 What is our pipeline good at?

Meaningful hierarchies:

Our method of grouping the generated questions (Section 3.5) produces hierarchies that clearly distinguish between general and specific questions; Figure 4 contains some hierarchies that support the positive results of our crowdsourced evaluation.

Top-k sampling:

Similar to prior work Fan et al. (2018); Holtzman et al. (2019), we notice that beam search often produces generic or repetitive beams (Table 5). Even though the top-k scheme always produces lower-probable questions than beam search, our filtering system prefers a top-k question 49.5% of the time.

“In 1942, Dodds enlisted in the US army and served as an anti aircraft gunner during World War II.”
In what year did the US army take place?
In what year did the US army take over?
In what year did the US army take place in the US?
What year was he enlisted?
When did he go to war?
When did he play as anti aircraft?
Table 5: Beam Search (B) vs Top-k sampling (T) for specific  question generation. Top-k candidates tend to be more diverse.

5.2 What kind of mistakes does it make?

We describe the various types of errors our model makes in this section, using the Paul Weston SQUASH  output in Figure 5 as a running example. Additionally, we list some modeling approaches we tried that did not work in Appendix C.

Reliance on a flawed answering system:

Our pipeline’s output is tied to the quality of the pretrained answering module, which both filters out questions and produces final answers. QuAC has long answer spans Choi et al. (2018) that cause low-precision predictions with extra information (e.g., “Who was born in Springfield?”). Additionally, the answering module occasionally swaps two named entities present in the paragraph.191919For instance in the sentence “The Carpenter siblings were born in New Haven, to Harold B. and Agnes R.” the model incorrectly answers the question “Who was born in New Haven?” as “Harold B. and Agnes R.”

Redundant information and lack of discourse:

In our system, each QA pair is generated independently of all the others. Hence, our outputs lack an inter-question discourse structure. Our system often produces a pair of redundant specific questions where the text of one question answers the other (e.g., “Who was born in Springfield?” vs. “Where was Weston born?”). These errors can likely be corrected by conditioning the generation module on previously-produced questions (or additional filtering); we leave this to future work.

Lack of world knowledge:

Our models lack commonsense knowledge (“How old was Weston when he was born?”) and can misinterpret polysemous words. Integrating pretrained contextualized embeddings Peters et al. (2018) into our pipeline is one potential solution.

Multiple general QA per paragraph:

Our system often produces more than one tree per paragraph, which is undesirable for short, focused paragraphs with a single topic sentence. To improve the user experience, it might be ideal to restrict the number of general questions we show per paragraph. While we found it difficult to generate general questions representative of entire paragraphs (Appendix C), a potential solution could involve identifying and generating questions from topic sentences.

Coreferences in general questions:

Many generated general questions contain coreferences due to contextual nature of the QuAC and CoQA training data (“How did he get into music?”). Potential solutions could involve either constrained decoding to avoid beams with anaphoric expressions or using the CorefNQG model of Du2018HarvestingPQ.

5.3 Which models did not work?

We present modelling approaches which did not work in Appendix C. This includes, i) end-to-end modelling to generate sequences of questions using QuAC, ii) span selection NER system, iii) generation of general questions representative of entire paragraphs, iv) answering system trained on the combination of QuAC, CoQA and SQuAD.

6 Related Work

Our work on SQUASH  is related to research in three broad areas: question generation, information retrieval and summarization.

Question Generation:

Our work builds upon neural question generation systems Du et al. (2017); Du and Cardie (2018). Our work conditions generation on specificity, similar to difficulty-conditioned question generation Gao et al. (2018). QA pair generation has previously been used for dataset creation Serban et al. (2016); Du and Cardie (2018). Joint modeling of question generation and answering has improved the performance of individual components Tang et al. (2017); Wang et al. (2017); Sachan and Xing (2018) and enabled visual dialog generation Jain et al. (2018).

Information Retrieval:

Our hierarchies are related to interactive retrieval setting Hardtke et al. (2009); Brandt et al. (2011) where similar webpages are grouped together. SQUASH  is also related to exploratory Marchionini (2006) and faceted search Yee et al. (2003).


Our work is related to query-focused summarization Dang (2005); Baumel et al. (2018) which conditions an output summary on an input query. Hierarchies have also been applied to summarization Christensen et al. (2014); Zhang et al. (2017); Tauchmann et al. (2018).

7 Future Work

While Section 5.2 focused on shortcomings in our modeling process and steps to fix them, this section focuses on broader guidelines for future work involving the SQUASH  format and its associated text generation task.

Evaluation of the SQUASH  format:

As discussed in Section 1, previous research shows support for the usefulness of hierarchies and QA in pedagogical applications. We did not directly evaluate this claim in the context of SQUASH , focusing instead on evaluating the quality of QA pairs and their hierarchies. Moving forward, careful user studies are needed to evaluate the efficacy of the SQUASH  format in pedagogical applications, which might be heavily domain-dependent; for example, a QA hierarchy for a research paper is likely to be more useful to an end user than a QA hierarchy for an online blog. An important caveat is the imperfection of modern text generation systems, which might cause users to prefer the original human-written document over a generated SQUASH  output. One possible solution is a three-way comparison between the original document, a human-written SQUASH ed document, and a system-generated output. For fair comparison, care should be taken to prevent experimenter bias while crowdsourcing QA hierarchies (e.g., by maintaining similar text complexity in the two human-written formats).

Collection of a SQUASH  dataset:

Besides measuring the usefulness of the QA hierarchies, a large dedicated dataset can help to facilitate end-to-end modeling. While asking human annotators to write full SQUASH ed documents will be expensive, a more practical option is to ask them to pair general and specific questions in our dataset to form meaningful hierarchies and write extra questions whenever no such pair exists.

QA budget and deeper specificity hierarchies:

In our work, we generate questions for every sentence and filter bad questions with fixed thresholds. An alternative formulation is an adaptive model dependent on a user-specified QA budget, akin to “target length” in summarization systems, which would allow end users to balance coverage and brevity themselves. A related modification is increasing the depth of the hierarchies. While two-level QA trees are likely sufficient for documents structured into short and focused paragraphs, deeper hierarchies can be useful for long unstructured chunks of text. Users can control this property via a “maximum children per QA node” hyperparameter, which along with the QA budget will determine the final depth of the hierarchy.

8 Conclusion

We propose SQUASH , a novel text generation task which converts a document into a hierarchy of QA pairs. We present and evaluate a system which leverages existing reading comprehension datasets to attempt solving this task. We believe SQUASH  is a challenging text generation task and we hope the community finds it useful to benchmark systems built for document understanding, question generation and question answering. Additionally, we hope that our specificity-labeled reading comprehension dataset is useful in other applications such as 1) finer control over question generation systems used in education applications, curiosity-driven chatbots and healthcare Du et al. (2017).


We thank the anonymous reviewers for their insightful comments. In addition, we thank Nader Akoury, Ari Kobren, Tu Vu and the other members of the UMass NLP group for helpful comments on earlier drafts of the paper and suggestions on the paper’s presentation. This work was supported in part by research awards from the Allen Institute for Artificial Intelligence and Adobe Research.


  • Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. International Conference on Learning Representations (ICLR).
  • Baumel et al. (2018) Tal Baumel, Matan Eyal, and Michael Elhadad. 2018. Query focused abstractive summarization: Incorporating query relevance, multi-document coverage, and summary length constraints into seq2seq models. arXiv preprint arXiv:1801.07704.
  • Brandt et al. (2011) Christina Brandt, Thorsten Joachims, Yisong Yue, and Jacob Bank. 2011. Dynamic ranked retrieval. In Proceedings of the fourth ACM international conference on Web search and data mining.
  • Buyukkokten et al. (2001) Orkut Buyukkokten, Hector Garcia-Molina, and Andreas Paepcke. 2001. Seeing the whole in parts: text summarization for web browsing on handheld devices. In Proceedings of the 10th international conference on World Wide Web.
  • Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. Quac: Question answering in context. In

    Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP)

  • Christensen et al. (2014) Janara Christensen, Stephen Soderland, Gagan Bansal, et al. 2014. Hierarchical summarization: Scaling up multi-document summarization. In Proc. Association for Computational Linguistics (ACL).
  • Dang (2005) Hoa Trang Dang. 2005. Overview of duc 2005. In Document Understanding Conferences.
  • Davenport et al. (1998) Thomas H Davenport, David W De Long, and Michael C Beers. 1998. Successful knowledge management projects. Sloan management review, 39(2):43–57.
  • Du and Cardie (2017) Xinya Du and Claire Cardie. 2017. Identifying where to focus in reading comprehension for neural question generation. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP).
  • Du and Cardie (2018) Xinya Du and Claire Cardie. 2018. Harvesting paragraph-level question-answer pairs from wikipedia. In Proc. Association for Computational Linguistics (ACL).
  • Du et al. (2017) Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Proc. Association for Computational Linguistics (ACL).
  • Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proc. Association for Computational Linguistics (ACL).
  • Gao et al. (2018) Yifan Gao, Jianan Wang, Lidong Bing, Irwin King, and Michael R Lyu. 2018. Difficulty controllable question generation for reading comprehension. arXiv preprint arXiv:1807.03586.
  • Gardner et al. (2018) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. Allennlp: A deep semantic natural language processing platform. arXiv preprint arXiv:1803.07640.
  • Gruber (2008) Tom Gruber. 2008. Collective knowledge systems: Where the social web meets the semantic web. Web semantics: science, services and agents on the World Wide Web, 6(1).
  • Hakkarainen and Sintonen (2002) Kai Hakkarainen and Matti Sintonen. 2002. The interrogative model of inquiry and computer-supported collaborative learning. Science & Education, 11(1):25–43.
  • Hardtke et al. (2009) David Hardtke, Mike Wertheim, and Mark Cramer. 2009. Demonstration of improved search result relevancy using real-time implicit relevance feedback. Understanding the User-Logging and Interpreting User Interactions in Information Search and Retrieval (UIIR-2009).
  • Heilman and Smith (2010) Michael Heilman and Noah A Smith. 2010. Good question! statistical ranking for question generation. In Proc. Conference of the North American Chapter of the Association for Computational Linguistics – Human Language Technologies (NAACL HLT).
  • Henderson et al. (2018) Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018.

    Deep reinforcement learning that matters.

    In Proc. Association for the Advancement of Artificial Intelligence (AAAI).
  • Hintikka (1981) Jaakko Hintikka. 1981. The logic of information-seeking dialogues: A model. Werner Becker and Wilhelm K. Essler Konzepte der Dialektik, pages 212–231.
  • Hintikka (1988) Jaakko Hintikka. 1988. What is the logic of experimental inquiry? Synthese, 74(2):173–190.
  • Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation.
  • Holtzman et al. (2018) Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learning to write with cooperative discriminators. In Proc. Association for Computational Linguistics (ACL).
  • Holtzman et al. (2019) Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
  • Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017.

    spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing.

    To appear.
  • Jain et al. (2018) Unnat Jain, Svetlana Lazebnik, and Alexander G Schwing. 2018. Two can play this game: visual dialog with discriminative question generation and answering. In

    Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR)

  • Kim et al. (2018) Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. 2018. Improving neural question generation using answer separation. arXiv preprint arXiv:1809.02393.
  • Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP).
  • Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proc. International Conference on Learning Representations (ICLR).
  • Landis and Koch (1977) J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174.
  • Lehnert (1978) Wendy G Lehnert. 1978. The process of question answering: A computer simulation of cognition, volume 978. Lawrence Erlbaum Hillsdale, NJ.
  • Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP).
  • Marchionini (2006) Gary Marchionini. 2006. Exploratory search: from finding to understanding. Communications of the ACM, 49(4):41–46.
  • Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. In NIPS 2017 Autodiff Workshop.
  • Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. Conference of the North American Chapter of the Association for Computational Linguistics – Human Language Technologies (NAACL HLT).
  • Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proc. Association for Computational Linguistics (ACL).
  • Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing, pages 2383–2392.
  • Reddy et al. (2018) Siva Reddy, Danqi Chen, and Christopher D. Manning. 2018. Coqa: A conversational question answering challenge. arXiv preprint arXiv:1808.07042.
  • Sachan and Xing (2018) Mrinmaya Sachan and Eric Xing. 2018. Self-training for jointly learning to ask and answer questions. In Proc. Conference of the North American Chapter of the Association for Computational Linguistics – Human Language Technologies (NAACL HLT).
  • Schank (1972) Roger C Schank. 1972. Conceptual dependency: A theory of natural language understanding. Cognitive psychology, 3(4):552–631.
  • See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proc. Association for Computational Linguistics (ACL).
  • Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. Association for Computational Linguistics (ACL).
  • Serban et al. (2016) Iulian Vlad Serban, Alberto García-Durán, Caglar Gulcehre, Sungjin Ahn, Sarath Chandar, Aaron Courville, and Yoshua Bengio. 2016.

    Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus.

    In Proc. Association for Computational Linguistics (ACL).
  • Stede and Schlangen (2004) Manfred Stede and David Schlangen. 2004. Information-seeking chat: Dialogues driven by topic-structure. In Proceedings of Catalog (the 8th workshop on the semantics and pragmatics of dialogue; SemDial04).
  • Tang et al. (2017) Duyu Tang, Nan Duan, Tao Qin, Zhao Yan, and Ming Zhou. 2017. Question answering and question generation as dual tasks. arXiv preprint arXiv:1706.02027.
  • Tauchmann et al. (2018) Christopher Tauchmann, Thomas Arnold, Andreas Hanselowski, Christian M Meyer, and Margot Mieskes. 2018. Beyond generic summarization: A multi-faceted hierarchical summarization corpus of large heterogeneous data. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC).
  • Wagner (2004) Christian Wagner. 2004. Wiki: A technology for conversational knowledge management and group collaboration. Communications of the association for information systems, 13(1):19.
  • Wagner and Bolloju (2005) Christian Wagner and Narasimha Bolloju. 2005. Supporting knowledge management in organizations with conversational technologies: Discussion forums, weblogs, and wikis. Journal of Database Management, 16(2).
  • Wang et al. (2017) Tong Wang, Xingdi Yuan, and Adam Trischler. 2017. A joint model for question answering and question generation. arXiv preprint arXiv:1706.01450.
  • Yatskar (2019) Mark Yatskar. 2019. A qualitative comparison of coqa, squad 2.0 and quac. Proc. Human Language Technology/Conference of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL).
  • Yee et al. (2003) Ka-Ping Yee, Kirsten Swearingen, Kevin Li, and Marti Hearst. 2003. Faceted metadata for image search and browsing. In Proceedings of the SIGCHI conference on Human factors in computing systems.
  • Zhang et al. (2017) Amy X. Zhang, Lea Verou, and David Karger. 2017. Wikum: Bridging discussion forums and wikis using recursive summarization. In Conference on Computer Supported Cooperative Work and Social Computing (CSCW).


Appendix A Question Classification Details

Confirming our intuition, Table 2 shows us that QuAC has the highest percentage of general  questions. On the other hand CoQA and SQuAD, which allowed the question-asker to look at the passage, are dominated by specific  questions. These findings are consistent with a comparison across the three datasets in yatskar2018qualitative. Interestingly, the average answer length for specific  questions in QuAC is 12 tokens, compared to 17 tokens for general  questions.
We provide the exact distribution of rule-labeled, hand-labeled and classifier-labeled questions in Table A1.

Appendix B Hyperparameters for Question Generation

Our question generation system consists of a two layer bidirectional LSTM encoder and a unidirectional LSTM decoder respectively. The LSTM hidden unit size in each direction and token embedding size is each set to 512. The class specificity embeddings size is 16. Embeddings are shared between the paragraph encoder and question decoder. All attention computations use a bilinear product (Luong et al., 2015). A dropout of 0.5 is used between LSTM layers. Models are trained using Adam Kingma and Ba (2014) with a learning rate of

, with a gradient clipping of 5.0 and minibatch size 32. Early stopping on validation perplexity is used to choose the best question generation model.

Appendix C What did not work?

End-to-End Sequential Generation.

We experimented with an end-to-end neural model which generated a sequence of questions given a sequence of answer spans. As training data, we leveraged the sequence IDs and follow-up information in the QuAC dataset, without specificity labels. We noticed that during decoding the model rarely attended over the history and often produced questions irrelevant to the context. A potential future direction would involve using the specificity labels for an end-to-end model.

Span Selection NER system.

As discussed in Section 3.1 and du2017identifying, we could frame answer span selection as a sequence labelling problem. We experimented with the NER system in AllenNLP (with ELMo embeddings) on the QuAC dataset, after the ground truth answer spans marked with BIO tags, after overlapping answers were merged together. We recorded low F1 scores of 33.3 and 15.6 on sentence-level and paragraph-level input respectively.

Paragraph-level question generation.

Our question generation model rarely generated general questions representative of the entire paragraph, even when we fed the entire paragraph as the answer span. We noticed that most general questions in our dataset were answered by one or two sentences in the paragraph.

Answering system trained on all datasets.

Recently, yatskar2018qualitative reported small improvements on the QuAC validation set by pre-training the BiDAF++ model on SQuAD 2.0 or CoQA. We tried combining the training data in all three datasets but achieved a validation F1 score of just 29.3 (compared to 50.2 after using just QuAC training data).

Dataset Size Rule Hand CNN
SQuAD 86.8k 30.5% 0.0% 69.5%
QuAC 65.2k 59.3% 1.5% 39.2%
CoQA 105.6k 57.1% 0.1% 42.8%
All 257.6k 48.7% 0.4% 50.9%
Table A1: Distribution of scheme adopted to classify questions in different datasets. “CNN” refers to the data-driven classifier. Roughly half the questions were classified using the rules described in Table 1.
Figure A1: Three SQUASH  outputs generated by our system, showcasing the strengths and weaknesses described in Section 5.