Neural Question Generation from Text: A Preliminary Study

04/06/2017 ∙ by Qingyu Zhou, et al. ∙ Beihang University Microsoft Harbin Institute of Technology 0

Automatic question generation aims to generate questions from a text passage where the generated questions can be answered by certain sub-spans of the given passage. Traditional methods mainly use rigid heuristic rules to transform a sentence into related questions. In this work, we propose to apply the neural encoder-decoder model to generate meaningful and diverse questions from natural language sentences. The encoder reads the input text and the answer position, to produce an answer-aware input representation, which is fed to the decoder to generate an answer focused question. We conduct a preliminary study on neural question generation from text with the SQuAD dataset, and the experiment results show that our method can produce fluent and diverse questions.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

Code Repositories

ConvAI-baseline

ConvAI baseline solution


view repo

NQG

Neural Question Generation from Text: A Preliminary Study


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Automatic question generation from natural language text aims to generate questions taking text as input, which has the potential value of education purpose (Heilman, 2011). As the reverse task of question answering, question generation also has the potential for providing a large scale corpus of question-answer pairs.

Previous works for question generation mainly use rigid heuristic rules to transform a sentence into related questions (Heilman, 2011; Chali and Hasan, 2015). However, these methods heavily rely on human-designed transformation and generation rules, which cannot be easily adopted to other domains. Instead of generating questions from texts, Serban et al. (2016)

proposed a neural network method to generate factoid questions from structured data.

In this work we conduct a preliminary study on question generation from text with neural networks, which is denoted as the Neural Question Generation (NQG) framework, to generate natural language questions from text without pre-defined rules. The Neural Question Generation framework extends the sequence-to-sequence models by enriching the encoder with answer and lexical features to generate answer focused questions. Concretely, the encoder reads not only the input sentence, but also the answer position indicator and lexical features. The answer position feature denotes the answer span in the input sentence, which is essential to generate answer relevant questions. The lexical features include part-of-speech (POS) and named entity (NER) tags to help produce better sentence encoding. Lastly, the decoder with attention mechanism (Bahdanau et al., 2015) generates an answer specific question of the sentence.

Large-scale manually annotated passage and question pairs play a crucial role in developing question generation systems. We propose to adapt the recently released Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016) as the training and development datasets for the question generation task. In SQuAD, the answers are labeled as subsequences in the given sentences by crowed sourcing, and it contains more than 100K questions which makes it feasible to train our neural network models. We conduct the experiments on SQuAD, and the experiment results show the neural network models can produce fluent and diverse questions from text.

2 Approach

In this section, we introduce the NQG framework, which consists of a feature-rich encoder and an attention-based decoder. Figure 1 provides an overview of our NQG framework.

Figure 1: Overview of the Neural Question Generation (NQG) framework.

2.1 Feature-Rich Encoder

In the NQG framework, we use Gated Recurrent Unit (GRU)

(Cho et al., 2014) to build the encoder. To capture more context information, we use bidirectional GRU (BiGRU) to read the inputs in both forward and backward orders. Inspired by Chen and Manning (2014); Nallapati et al. (2016)

, the BiGRU encoder not only reads the sentence words, but also handcrafted features, to produce a sequence of word-and-feature vectors. We concatenate the word vector, lexical feature embedding vectors and answer position indicator embedding vector as the input of BiGRU encoder. Concretely, the BiGRU encoder reads the concatenated sentence word vector, lexical features, and answer position feature,

, to produce two sequences of hidden vectors, i.e., the forward sequence and the backward sequence . Lastly, the output sequence of the encoder is the concatenation of the two sequences, i.e., .

Answer Position Feature

To generate a question with respect to a specific answer in a sentence, we propose using answer position feature to locate the target answer. In this work, the BIO tagging scheme is used to label the position of a target answer. In this scheme, tag B denotes the start of an answer, tag I continues the answer and tag O marks words that do not form part of an answer. The BIO tags of answer position are embedded to real-valued vectors throu and fed to the feature-rich encoder. With the BIO tagging feature, the answer position is encoded to the hidden vectors and used to generate answer focused questions.

Lexical Features

Besides the sentence words, we also feed other lexical features to the encoder. To encode more linguistic information, we select word case, POS and NER tags as the lexical features. As an intermediate layer of full parsing, POS tag feature is important in many NLP tasks, such as information extraction and dependency parsing (Manning et al., 1999). Considering that SQuAD is constructed using Wikipedia articles, which contain lots of named entities, we add NER feature to help detecting them.

2.2 Attention-Based Decoder

We employ an attention-based GRU decoder to decode the sentence and answer information to generate questions. At decoding time step , the GRU decoder reads the previous word embedding and context vector to compute the new hidden state . We use a linear layer with the last backward encoder hidden state to initialize the decoder GRU hidden state. The context vector for current time step is computed through the concatenate attention mechanism (Luong et al., 2015), which matches the current decoder state with each encoder hidden state to get an importance score. The importance scores are then normalized to get the current context vector by weighted sum:

(1)
(2)
(3)
(4)
(5)

We then combine the previous word embedding , the current context vector , and the decoder state to get the readout state . The readout state is passed through a maxout hidden layer (Goodfellow et al., 2013)

to predict the next word with a softmax layer over the decoder vocabulary:

(6)
(7)
(8)

where is a -dimensional vector.

2.3 Copy Mechanism

To deal with the rare and unknown words problem, Gulcehre et al. (2016) propose using pointing mechanism to copy rare words from source sentence. We apply this pointing method in our NQG system. When decoding word , the copy switch takes current decoder state and context vector

as input and generates the probability

of copying a word from source sentence:

(9)

where

is sigmoid function. We reuse the attention probability in equation

4 to decide which word to copy.

3 Experiments and Results

We use the SQuAD dataset as our training data. SQuAD is composed of more than 100K questions posed by crowd workers on 536 Wikipedia articles. We extract sentence-answer-question triples to build the training, development and test sets111We re-distribute the processed data split and PCFG-Trans baseline code at an anonymous url for blind review. . Since the test set is not publicly available, we randomly halve the development set to construct the new development and test sets. The extracted training, development and test sets contain 86,635, 8,965 and 8,964 triples respectively. We introduce the implementation details in the appendix.

We conduct several experiments and ablation tests as follows:

[noitemsep]

PCFG-Trans

The rule-based system

1 modified on the code released by Heilman (2011). We modified the code so that it can generate question based on a given word span.

s2s+att

We implement a seq2seq with attention as the baseline method.

NQG

We extend the s2s+att with our feature-rich encoder to build the NQG system.

NQG+

Based on NQG, we incorporate copy mechanism to deal with rare words problem.

NQG+Pretrain

Based on NQG+, we initialize the word embedding matrix with pre-trained GloVe (Pennington et al., 2014) vectors.

NQG+STshare

Based on NQG+, we make the encoder and decoder share the same embedding matrix.

NQG++

Based on NQG+, we use both pre-train word embedding and STshare methods, to further improve the performance.

NQGAnswer

Ablation test, the answer position indicator is removed from NQG model.

NQGPOS

Ablation test, the POS tag feature is removed from NQG model.

NQGNER

Ablation test, the NER feature is removed from NQG model.

NQGCase

Ablation test, the word case feature is removed from NQG model.

3.1 Results and Analysis

We report BLEU-4 score (Papineni et al., 2002)

as the evaluation metric of our NQG system.

Model Dev set Test set
PCFG-Trans 9.28 9.31
s2s+att 3.01 3.06
NQG 10.06 10.13
NQG+ 12.30 12.18
NQG+Pretrain 12.80 12.69
NQG+STshare 12.92 12.80
NQG++ 13.27 13.29
NQGAnswer 2.79 2.98
NQGPOS 9.83 9.87
NQGNER 9.50 9.29
NQGCase 9.91 9.89
Table 1: BLEU evaluation scores of baseline methods, different NQG framework configurations and some ablation tests.

Table 1 shows the BLEU-4 scores of different settings. We report the beam search results on both development and test sets. Our NQG framework outperforms the PCFG-Trans and s2s+att baselines by a large margin. This shows that the lexical features and answer position indicator can benefit the question generation. With the help of copy mechanism, NQG+ has a 2.05 BLEU improvement since it solves the rare words problem. The extended version, NQG++, has 1.11 BLEU score gain over NQG+, which shows that initializing with pre-trained word vectors and sharing them between encoder and decoder help learn better word representation.

Human Evaluation

We evaluate the PCFG-Trans baseline and NQG++ with human judges. The rating scheme is, Good (3) - The question is meaningful and matches the sentence and answer very well; Borderline (2) - The question matches the sentence and answer, more or less; Bad (1) - The question either does not make sense or matches the sentence and answer. We provide more detailed rating examples in the supplementary material. Three human raters labeled 200 questions sampled from the test set to judge if the generated question matches the given sentence and answer span. The inter-rater aggreement is measured with Fleiss’ kappa (Fleiss, 1971).

Model AvgScore Fleiss’ kappa
PCFG-Trans 1.42 0.50
NQG++ 2.18 0.46
Table 2: Human evaluation results.

Table 2 reports the human judge results. The kappa scores show a moderate agreement between the human raters. Our NQG++ outperforms the PCFG-Trans baseline by 0.76 score, which shows that the questions generated by NQG++ are more related to the given sentence and answer span.

Ablation Test

The answer position indicator, as expected, plays a crucial role in answer focused question generation as shown in the NQGAnswer ablation test. Without it, the performance drops terribly since the decoder has no information about the answer subsequence.

Ablation tests, NQGCase, NQGPOS and NQGNER, show that word case, POS and NER tag features contributes to question generation.

Case Study

Table 3 provides three examples generated by NQG++. The words with underline are the target answers. These three examples are with different question types, namely WHEN, WHAT and WHO respectively. It can be observed that the decoder can ‘copy’ spans from input sentences to generate the questions. Besides the underlined words , other meaningful spans can also be used as answer to generate correct answer focused questions.

I: in 1226 , immediately after returning from the west , genghis khan began a retaliatory attack on the tanguts .
G: in which year did genghis khan strike against the tanguts ?
O: in what year did genghis khan begin a retaliatory attack on the tanguts ?
I: in week 10 , manning suffered a partial tear of the plantar fasciitis in his left foot .
G: in the 10th week of the 2015 season , what injury was peyton manning dealing with ?
O: what did manning suffer in his left foot ?
I: like the lombardi trophy , the “ 50 ” will be designed by tiffany & co. .
G: who designed the vince lombardi trophy ?
O: who designed the lombardi trophy ?
Table 3: Examples of generated questions, I is the input sentence, G is the gold question and O is the NQG++ generated question. The underlined words are the target answers.

Type of Generated Questions

Following Wang and Jiang (2016)

, we classify the questions into different types, i.e., WHAT, HOW, WHO, WHEN, WHICH, WHERE, WHY and OTHER.

222We treat questions ‘what country’, ‘what place’ and so on as WHERE type questions. Similarly, questions containing ‘what time’, ‘what year’ and so forth are counted as WHEN type questions.

We evaluate the precision and recall of each question types. Figure

2 provides the precision and recall metrics of different question types. The precision and recall of a question type are defined as:

precision(T) (10)
recall(T) (11)
Figure 2: Precision and recall of question types.

For the majority question types, WHAT, HOW, WHO and WHEN types, our NQG++ model performs well for both precision and recall. For type WHICH, it can be observed that neither precision nor recall are acceptable. Two reasons may cause this: a) some WHICH-type questions can be asked in other manners, e.g., ‘which team’ can be replaced with ‘who’; b) WHICH-type questions account for about 7.2% in training data, which may not be sufficient to learn to generate this type of questions. The same reason can also affect the precision and recall of WHY-type questions.

4 Conclusion and Future Work

In this paper we conduct a preliminary study of natural language question generation with neural network models. We propose to apply neural encoder-decoder model to generate answer focused questions based on natural language sentences. The proposed approach uses a feature-rich encoder to encode answer position, POS and NER tag information. Experiments show the effectiveness of our NQG method. In future work, we would like to investigate whether the automatically generated questions can help to improve question answering systems.

References

  • Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of 3rd International Conference for Learning Representations. San Diego.
  • Chali and Hasan (2015) Yllias Chali and Sadid A. Hasan. 2015. Towards topic-to-question generation. Comput. Linguist. 41(1):1–20.
  • Chen and Manning (2014) Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of EMNLP 2014. Association for Computational Linguistics, Doha, Qatar, pages 740–750.
  • Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of EMNLP 2014. Doha, Qatar, pages 1724–1734.
  • Fleiss (1971) Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76(5):378.
  • Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Aistats. volume 9, pages 249–256.
  • Goodfellow et al. (2013) Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron C Courville, and Yoshua Bengio. 2013. Maxout networks. ICML (3) 28:1319–1327.
  • Gulcehre et al. (2016) Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 140–149.
  • Heilman (2011) Michael Heilman. 2011. Automatic factual question generation from text. Ph.D. thesis, Carnegie Mellon University.
  • Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of 3rd International Conference for Learning Representations. San Diego.
  • Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of EMNLP 2015. Association for Computational Linguistics, Lisbon, Portugal, pages 1412–1421.
  • Manning et al. (1999) Christopher D Manning, Hinrich Schütze, et al. 1999.

    Foundations of statistical natural language processing

    , volume 999.
    MIT Press.
  • Manning et al. (2014) Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations. pages 55–60.
  • Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Ça glar Gulçehre, and Bing Xiang. 2016.

    Abstractive text summarization using sequence-to-sequence rnns and beyond.

    In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning.
  • Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pages 311–318.
  • Pascanu et al. (2013) Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013.

    On the difficulty of training recurrent neural networks.

    ICML (3) 28:1310–1318.
  • Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP). pages 1532–1543.
  • Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 .
  • Serban et al. (2016) Iulian Vlad Serban, Alberto García-Durán, Caglar Gulcehre, Sungjin Ahn, Sarath Chandar, Aaron Courville, and Yoshua Bengio. 2016. Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 588–598.
  • Srivastava et al. (2014) Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting.

    Journal of Machine Learning Research

    15(1):1929–1958.
  • Wang and Jiang (2016) Shuohang Wang and Jing Jiang. 2016. Machine comprehension using match-lstm and answer pointer. arXiv preprint arXiv:1608.07905 .

Appendix A Implementation Details

a.1 Model Parameters

We use the same vocabulary for both encoder and decoder. The vocabulary is collected from the training data and we keep the top 20,000 frequent words. We set the word embedding size to 300 and all GRU hidden state sizes to 512. The lexical and answer position features are embedded to 32-dimensional vectors. We use dropout (Srivastava et al., 2014) with probability . During testing, we use beam search with beam size 12.

a.2 Lexical Feature Annotation

We use Stanford CoreNLP v3.7.0 (Manning et al., 2014) to annotate POS and NER tags in sentences with its default configuration and pre-trained models.

a.3 Model Training

We initialize model parameters randomly using a Gaussian distribution with Xavier scheme

(Glorot and Bengio, 2010). We use a combination of Adam (Kingma and Ba, 2015)

and simple SGD as our the optimizing algorithms. The training is separated into two phases, the first phase is optimizing the loss function with Adam and the second is with simple SGD. For the Adam optimizer, we set the learning rate

, two momentum parameters and respectively, and . We use Adam optimizer until the BLEU score on the development set drops for six consecutive tests (we test the BLEU score on the development set for every 1,000 batches). Then we switch to a simple SGD optimizer with initial learning rate

and halve it if the BLEU score on the development set drops for twelve consecutive tests. We also apply gradient clipping

(Pascanu et al., 2013) with range for both Adam and SGD phases. To both speed up the training and converge quickly, we use mini-batch size 64 by grid search.

Appendix B Human Evaluation Examples

We evaluate the PCFG-Trans baseline and NQG++ with human judges. The rating scheme is provided in Table 4.

Score Rating scheme
3: Good The question is meaningful and matches the sentence and answer very well
2: Borderline The question matches the sentence and answer, more or less
1: Bad The question either does not make sense or matches the sentence and answer
Table 4: Human rating scheme.

The human judges are asked to label the generated questions if they match the given sentence and answer span according to the rating scheme and examples. We provide some example questions with different scores in Table 5. For the first score 3 example, the question makes sense and the target answer “reason” can be used to answer it given the input sentence. For the second score 2 example, the question is inadequate for answering the sentence since the answer is about prime number. However, given the sentence, a reasonable person will give the targeted answer of the question. For the third score 1 example, the question is totally wrong given the sentence and answer.

Score Sentence and generated question
3 I: -lsb- … -rsb- for reason is the greatest enemy that faith has ; it never comes to the aid of spiritual things . ”
O: what is the biggest enemy that faith has have ?
2 I: in all other rows -lrb- a = 1 , 2 , 4 , 5 , 7 , and 8 -rrb- there are infinitely many prime numbers .
O: how many numbers are in all other rows ?
1 I: while genghis khan never conquered all of china , his grandson kublai khan completed that conquest and established the yuan dynasty that is often credited with re-uniting china .
O: who did kublai khan defeat that conquered all of china ?
Table 5: Human rating scheme examples, I is input, O is output, the underlined words are target answers.