Coherent Dialogue with Attention-based Language Models

11/21/2016 ∙ by Hongyuan Mei, et al. ∙ University of North Carolina at Chapel Hill Johns Hopkins University Toyota Technological Institute at Chicago 0

We model coherent conversation continuation via RNN-based dialogue models equipped with a dynamic attention mechanism. Our attention-RNN language model dynamically increases the scope of attention on the history as the conversation continues, as opposed to standard attention (or alignment) models with a fixed input scope in a sequence-to-sequence model. This allows each generated word to be associated with the most relevant words in its corresponding conversation history. We evaluate the model on two popular dialogue datasets, the open-domain MovieTriples dataset and the closed-domain Ubuntu Troubleshoot dataset, and achieve significant improvements over the state-of-the-art and baselines on several metrics, including complementary diversity-based metrics, human evaluation, and qualitative visualizations. We also show that a vanilla RNN with dynamic attention outperforms more complex memory models (e.g., LSTM and GRU) by allowing for flexible, long-distance memory. We promote further coherence via topic modeling-based reranking.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Automatic conversational models [Winograd1971], also known as dialogue systems, are of great importance to a large variety of applications, ranging from open-domain entertaining chatbots to goal-oriented technical support agents. An increasing amount of research has recently been done to build purely data-driven dialogue systems that learn from large corpora of human-to-human conversations, without using hand-crafted rules or templates. While most work in this area formulates dialogue modeling in a sequence-to-sequence framework (similar to machine translation) [Ritter, Cherry, and Dolan2011, Shang, Lu, and Li2015, Vinyals and Le2015, Sordoni et al.2015, Li et al.2016a, Dušek and Jurčíček2016], some more recent work [Serban et al.2016, Luan, Ji, and Ostendorf2016] instead trains a language model over the entire dialogue as one single sequence. In our work, we empirically demonstrate that a language model is better suited to dialogue modeling, as it learns how the conversation evolves as information progresses. Sequence-to-sequence models, on the other hand, learn only how the most recent dialogue response is generated. Such models are better suited to converting the same information from one modality to another, e.g., in machine translation and image captioning.

We improve the coherence of such neural dialogue language models by developing a generative dynamic attention mechanism that allows each generated word to choose which related words it wants to align to in the increasing conversation history (including the previous words in the response being generated). Neural attention (or alignment) has proven very successful for various sequence-to-sequence tasks by associating salient items in the source sequence with the generated item in the target sequence [Mnih et al.2014, Bahdanau, Cho, and Bengio2015, Xu et al.2015, Mei, Bansal, and Walter2016a, Parikh et al.2016]

. However, such attention models are limited to a fixed scope of history, corresponding to the input source sequence. In contrast, we introduce a dynamic attention mechanism to a recurrent neural network (RNN) language model in which the scope of attention increases as the recurrence operation progresses from the start through the end of the conversation.

((a)) RNN seq2seq (encoder-decoder) model
((b)) RNN language model
((c)) Attention seq2seq (encoder-decoder) model
((d)) Attention language model
Figure 1: Comparing RNN language models to RNN sequence-to-sequence model, with and without attention.

The dynamic attention model promotes coherence of the generated dialogue responses (continuations) by favoring the generation of words that have syntactic or semantic associations with salient words in the conversation history. Our simple model shows significant improvements over state-of-the-art models and baselines on several metrics (including complementary diversity-based metrics, human evaluation, and qualitative visualizations) for two benchmark datasets, the open-domain MovieTriples and closed-domain Ubuntu Troubleshoot datasets. Our vanilla RNN model with dynamic attention outperforms more complex memory models (e.g., LSTM and GRU) by allowing for long-distance and flexible memory. We also present several visualizations to intuitively understand what the attention model is learning. Finally, we also explore a complementary LDA-based method to re-rank the outputs of the soft alignment-based coherence method, further improving performance on the evaluation benchmarks.

2 Related Work

A great deal of attention has been paid to developing data-driven methods for natural language dialogue generation. Conventional statistical approaches tend to rely extensively on hand-crafted rules and templates, require interaction with humans or simulated users to optimize parameters, or produce conversation responses in an information retrieval fashion. Such properties prevent training on the large human conversational corpora that are becoming increasingly available, or fail to produce novel natural language responses.

ritter-11-gen (ritter-11-gen) formulate dialogue response generation as a statistical phrase-based machine translation problem, which requires no explicit hand-crafted rules. The recent success of RNNs in statistical machine translation [Sutskever, Vinyals, and Lee2014, Bahdanau, Cho, and Bengio2015] has inspired the application of such models to the field of dialogue modeling. vinyals-15-conv (vinyals-15-conv) and shang-15-conv (shang-15-conv) employ an RNN to generate responses in human-to-human conversations by treating the conversation history as one single temporally ordered sequence. In such models, the distant relevant context in the history is difficult to recall. Some efforts have been made to overcome this limitation. sordoni-15 (sordoni-15) separately encode the most recent message and all the previous context using a bag-of-words representation, which is decoded using an RNN. This approach equates the distance of each word in the generated output to all the words in the conversation history, but loses the temporal information of the history. serban-16 (serban-16) design a hierarchical model that stacks an utterance-level RNN on a token-level RNN, where the utterance-level RNN reduces the number of computational steps between utterances. wen-15-semantic (wen-15-semantic) and wen-16-multi (wen-16-multi) improve spoken dialog systems via multi-domain and semantically conditioned neural networks on dialog act representations and explicit slot-value formulations.

Our work explores the ability of recurrent neural network language models [Bengio et al.2003, Mikolov2010] to interpret and generate natural language conversations while still maintaining a relatively simple architecture. We show that a language model approach outperforms the sequence-to-sequence model at dialogue modeling. Recently, tran-16 (tran-16) demonstrated that the neural attention mechanism can improve the effectiveness of a neural language model. We propose an attention-based neural language model for dialogue modeling that learns how a conversation evolves as a whole, rather than only how the most recent response is generated, and that also reduces the number of computations between the current recurrence step and the distant relevant context in the conversation history.

The attention mechanism in our model has the additional benefit of favoring words that have semantic association with salient words in the conversation history, which promotes the coherence of the topics in the continued dialogue. This is important when conversation participants inherently want to maintain the topic of the discussion. Some past studies have equated coherence with propositional consistency [Goldberg1983], while others see it as a summary impression [Sanders1983]. Our work falls in the category of viewing coherence as topic continuity [Crow1983, Sigman1983]. Similar objectives, i.e., generating dialogue responses with certain properties, have been addressed recently, such as promoting response diversity [Li et al.2016a], enhancing personal consistency [Li et al.2016b], and improving specificity [Yao et al.2016]. Concurrent with this work, luan-16 (luan-16) improve topic consistency by feeding into the model the learned LDA-based topic representations. We show that the simple attention neural language model significantly outperforms such a design. Furthermore, we suggest an LDA-based re-ranker complementary to soft neural attention that further promotes topic coherence.

3 The Model

In this section, we introduce an attention-RNN dialogue model and compare it to the architectures of basic RNN and sequence-to-sequence models.

3.1 RNN Seq2Seq and Language Models

Recurrent neural networks have been successfully used both in sequence-to-sequence models (RNN-Seq2Seq, Fig. 0(a)[Sutskever, Vinyals, and Lee2014] and in language models (RNN-LM, Fig. 0(b)[Bengio et al.2003, Mikolov2010]. We first discuss language models for dialogue, which is the primary focus of our work, then briefly introduce the sequence-to-sequence model, and lastly discuss the use of attention methods in both models.

The RNN-LM models a sentence as a sequence of tokens with a recurrence function

(1)

and an output (softmax) function

(2)

where the recurrent hidden state encodes all the tokens up to

and is used to compute the probability of generating

as the next token from the vocabulary .

The functions and are typically defined as

(3a)
(3b)

where is the recurrence matrix, is a column of word embedding matrix that corresponds to , projects word embedding into the space of the same dimension as the hidden units, and

is the output word embedding matrix with column vector

corresponding to .

We train the RNN-LM, i.e, estimate the parameters

, , and , by maximizing the log-likelihood on a set of natural language training sentences of size

(4)

Since the entire architecture is differentiable, the objective can be optimized by back-propagation.

When dialogue is formulated as a sequence-to-sequence task, the RNN-Seq2Seq model can be used in order to predict a target sequence given an input source sequence . In such settings, an encoder RNN represents the input as a sequence of hidden states , and a separate decoder RNN then predicts the target sequence token-by-token given the encoder hidden states .

3.2 Attention in RNN-Seq2Seq Models

There are several ways by which to integrate the sequence of hidden states in the decoder RNN. An attention mechanism (Fig. 0(c)) has proven to be particularly effective for various related tasks in machine translation, image caption synthesis, and language understanding [Mnih et al.2014, Bahdanau, Cho, and Bengio2015, Xu et al.2015, Mei, Bansal, and Walter2016a].

The attention module takes as input the encoder hidden state sequence and the decoder hidden state at each step , and returns a context vector computed as a weighted average of encoder hidden states

(5a)
(5b)
(5c)

where parameters , , and are jointly learned with the other model parameters. The context vector is then used as an extra input to the decoder RNN at step together with to predict the next token .

3.3 Attention in RNN-LM

We develop an attention-RNN language model (A-RNN-LM) as illustrated in Figure 0(d), and describe how it can be used in the context of dialogue modeling. We then describe its advantages compared to the use of attention in sequence-to-sequence models.

As with the RNN-LM, the model first encodes the input into a sequence of hidden states up to word (Eqn. 1). Given a representation of tokens up to (which we define shortly), the attention module computes the context vector at step as a weighted average of

(6a)
(6b)
(6c)

We then use the context vector together with the hidden state to predict the output at time

(7a)
(7b)

where and project and , respectively, into the same space of dimension .

There are multiple benefits of using an attention-RNN language model for dialogue, which are empirically supported by our experimental results. First, a complete dialogue is usually composed of multiple turns. A language model over the entire dialogue is expected to better learn how a conversation evolves as a whole, unlike a sequence-to-sequence model, which only learns how the most recent response is generated and is better suited to translation-style tasks that transform the same information from one modality to another. Second, compared to LSTM models, an attention-based RNN-LM also allows for gapped context and a flexible combination of conversation history for every individual generated token, while maintaining low model complexity. Third, attention models yield interpretable results—we visualize the learned attention weights, showing how attention chooses the salient words from the dialogue history that are important for generating each new word. Such a visualization is typically harder for the hidden states and gates of conventional LSTM and RNN language models.

With an attention mechanism, there are multiple options for defining the token representation . The original attention model introduced by bahdanau-15 (bahdanau-15) uses the hidden units as the token representations . Recent work [Mei, Bansal, and Walter2016a, Mei, Bansal, and Walter2016b] has demonstrated that performance can be improved by using multiple abstractions of the input, e.g., , which is what we use in this work.

3.4 LDA-based Re-Ranking

While the trained attention-RNN dialogue model generates a natural language continuation of a conversation while maintaining topic concentration by token association, some dialogue-level topic-supervision can help to encourage generations that are more topic-aware. Such supervision is not commonly available, and we use unsupervised methods to learn document-level latent topics. We employ the learned topic model to select the best continuation based on document-level topic-matching.

We choose Latent Dirichlet Allocation (LDA) [Blei, Ng, and Jordan2003, Blei and Lafferty2009] due to its demonstrated ability to learn a distribution over latent topics given a collection of documents. This generative model assumes documents arise from topics, each of which is defined as a distribution over a fixed vocabulary of terms, which forms a graphical structure that can be learned from the training data. The topic representation of a (possibly unseen) dialogue can then be estimated with the learned topic structure as .

Given a set of generated continuations for each unseen dialogue , the topic representations of the dialogue and its continuations are and , respectively. We employ a matching score to compute the similarity between and each . In the end, a weighted score is computed as , where and is the conditional log-likelihood of the continuation . The hyper-parameters and are tuned on a development set.

Concurrent with our work, luan-16 (luan-16) use learned topic representations of the given conversation as an extra feature in a language model to enhance the topic coherence of the generation. As we show in the Results and Analysis section, our model significantly outperforms this approach.

4 Experimental Setup

[align=center]A: person , but this time i got the money . R: if you give it to me , i might forget i found you . A: i ’ ll give it to you . [align=center]A: person , but this time i got the money . R: if you give it to me , i might forget i found you . A: i ’ ll give it to you .
((a)) MovieTriples
[align=center]A: i have really choppy streaming video in 9.10 , any way to fix that ? A: any way to fix choppy streaming video in 9.10 ? R: what kind of video R: what video card do you have ? [align=center]A: i have really choppy streaming video in 9.10 , any way to fix that ? A: any way to fix choppy streaming video in 9.10 ? R: what kind of video R: what video card do you have ?
((b)) Ubuntu Troubleshoot
Figure 2: A visualization of attention on the att:movie MovieTriples and att:ubuntu Ubuntu Troubleshooting datasets, showing which words in the conversation history are being aligned to, for each generated response word. Shaded intervals indicate the strength with which the corresponding words in the conversation history and response are attend to when generating the bolded word in the response. We show this for two generated words in the same response (left and right column).

4.1 Dataset

We train and evaluate the models on two large natural language dialogue datasets, MovieTriples (pre-processed by serban-16 (serban-16)) and Ubuntu Troubleshoot (pre-processed by luan-16 (luan-16)). The dialogue within each of these datasets consists of a sequence of utterances (turns), each of which is a sequence of tokens (words).111Following luan-16 (luan-16), we randomly sample nine utterances as negative examples of the last utterance for each conversation in Ubuntu Troubleshoot for the development set. The accompanying supplementary material provides the statistics for these two datasets.

4.2 Evaluation Metrics

For the sake of comparison, we closely follow previous work and adopt several standard (and complementary) evaluation metrics: perplexity (PPL), word error rate (WER), recall@N, BLEU, and diversity-based Distinct-1. We provide further discussion of the various metrics and their advantages in the supplementary material. On the

MovieTriples dataset, we use PPL and WER, as is done in previous work. Following serban-16 (serban-16), we adopt two versions for each metric: i) PPL as the word-level perplexity over the entire dialogue conversation; ii) PPL@L as the word-level perplexity over the last utterance of the conversation; iii) WER; and iv) WER@L (defined similarly).

On the Ubuntu dataset, we follow previous work and use PPL and recall@N. Recall@N [Manning et al.2008] evaluates a model by measuring how often the model ranks the correct dialogue continuation within top-N given candidates. Additionally, we also employ the BLEU score [Papineni et al.2001] to evaluate the quality of the generations produced by the models. Following luan-16 (luan-16), we perform model selection using PPL on the development set, and perform the evaluation on the test set using the other metrics. We also present evaluation using the Distinct-1 metric (proposed by li-16-diversity (li-16-diversity)) to measure the ability of the A-RNN to promote diversity in the generations, because typical neural dialogue models generate generic, safe responses (technically appropriate but not informative, e.g., “I don t know”). Finally, we also present a preliminary human evaluation.

4.3 Training Details

For the MovieTriples dataset, we follow the same procedure as serban-16 (serban-16) and first pretrain on the large Q-A SubTitle dataset  [Ameixa et al.2014], which contains M question-answer pairs from which we randomly sample pairs as the held-out set, and then fine-tune on the target MovieTriples dataset. We perform early-stopping according to the PPL score on the held-out set. We train the models for both the MovieTriples and Ubuntu Troubleshoot datasets using Adam [Kingma and Ba2015] for optimization in RNN back-propagation. The accompanying supplementary material provides additional training details, including the hyper-parameter settings.

5 Results and Analysis

5.1 Primary Dialogue Modeling Results

In this section, we compare the performance on several metrics of our attention-based RNN-LM with RNN baselines and state-of-the-art models on the two benchmark datasets. Table 1 reports PPL and WER results on the MovieTriples test set, while Table 2 compares different models on Ubuntu Troubleshoot in terms of PPL on the development set and recall@N () on the test set (following what previous work reports). In the tables, RNN is the plain vanilla RNN language model (RNN-LM), as defined in The Model section, and LSTM is an LSTM-RNN language model, i.e., an RNN-LM with LSTM memory units. A-RNN refers to our main model as defined in the Attention in RNN-LM section. HRED in Table 1 is the hierarchical neural dialogue model proposed by serban-16 (serban-16).222We compare to their best-performing model version, that adopts bidirectional gated-unit RNN (GRU). LDA-CONV in Table 2 is proposed by luan-16 (luan-16), which integrates learned LDA-topic-proportions into an LSTM language model in order to promote topic-concentration in the generations. Both tables demonstrate that the attention-RNN-LM (A-RNN) model achieves the best results reported to-date on these datasets in terms all evaluation metrics. It improves the ability of an RNN-LM to model continuous dialogue conversations, while keeping the model architecture simple.

Model PPL PPL@L WER WER@L
RNN
HRED
A-RNN
Table 1: Results on the MovieTriples test set. The HRED results are from Serban et al. (2016).
Model PPL recall@ recall@
RNN
LSTM
LDA-CONV
A-RNN
Table 2: Ubuntu Troubleshoot PPL and recall@N, with LSTM and LDA-CONV results from Luan et al. (2016).
PPL MovieTriples Ubuntu Troubleshoot
RNN-Seq2Seq
RNN-LM
Table 3: RNN-LM vs. RNN-Seq2Seq
Model dev BLEU test BLEU
RNN
A-RNN
T-A-RNN
A-RNN-RR
T-A-RNN-RR
Table 4: BLEU score on Ubuntu Troubleshoot
History: A: person , but this time i got the money .
R: if you give it to me , i might forget i found you .
A-RNN: A: i ’ ll give it to you .
RNN: A: i don ’ t know .
Reference: A: i don ’ t have it with me . tell person .
((a)) MovieTriples
History: A: i have really choppy streaming video in 9.10, any way to fix that?
A: any way to fix choppy streaming video in 9.10 ?
R: what kind of video
A-RNN: A: what video card do you have ?
RNN: A: what are you trying to do ?
Reference: A: what cpu do you have
((b)) Ubuntu Troubleshoot
Figure 3: Examples of responses generated by the A-RNN.
Distinct-1 MovieTriples Ubuntu Troubleshoot
RNN
A-RNN
Table 5: Generation Diversity Results: A-RNN vs. RNN

We also evaluate the effectiveness of the RNN-LM and RNN-Seq2Seq models on both the MovieTriples and Ubuntu Troubleshoot development sets. As shown in Table 3, the RNN language model yields lower perplexity than the RNN sequence-to-sequence model on both datasets. Hence, we present all primary results on our primary A-RNN attention-based RNN language model.333Experiments also demonstrate significant improvements for the Attention-RNN-LM over the Attention-RNN-Seq2Seq.

5.2 Generation Diversity Results

Next, we investigate the ability of the A-RNN to promote diversity in the generations, compared to that of the vanilla RNN using the Distinct-1 metric proposed by li-16-diversity (li-16-diversity). Distinct-1 is computed as the number of distinct unigrams in the generation scaled by the total number of generated tokens. Table 5 shows that our attention-based RNN language model (A-RNN) yields much more diversity in its generations as compared to the vanilla RNN baseline.

5.3 Topic Coherence Results

Next, we investigate the ability of the different models to promote topic coherence in the generations in terms of BLEU score. In addition to the RNN and A-RNN models, we consider T-A-RNN, a method that incorporates LDA-based topic information into an A-RNN model, following the approach of luan-16 (luan-16). We also evaluate our LDA-based re-ranker, A-RNN-RR, which re-ranks according to the score , where we compute the log-likelihood based upon a trained A-RNN-M model and validate the weight on the development set. We also consider a method that combines the T-A-RNN model with an LDA-based re-ranker (T-A-RNN-RR).444Since luan-16 (luan-16) do not publish BLEU scores or implementations of their models, we can not compare with LDA-CONV on BLEU. Instead, we demonstrate the effect of adding the key component of LDA-CONV on top of the A-RNN. Table 4 reports the resulting BLEU scores for each of these methods on the development and test sets from the Ubuntu Troubleshoot dataset. We make the following observations based upon these results: (1) The A-RNN performs substantially better than the RNN with regards to BLEU; (2) using our LDA-based re-ranker further improves the performance by a significant amount (A-RNN v.s. A-RNN-RR); (3) as opposed to our LDA-based re-ranker, adopting the LDA design of luan-16 (luan-16) only yields marginal improvements on the development set, but does not generalize well to the test set (A-RNN v.s. T-A-RNN and A-RNN-RR v.s. T-A-RNN-RR). Also, our LDA re-ranker results in substantial improvements even on top of their topic-based model (T-A-RNN v.s. T-A-RNN-RR).

5.4 Preliminary Human Evaluation

In addition to multiple automatic metrics, we also report a preliminary human evaluation. On each dataset, we manually evaluate the generations of both the A-RNN and RNN models on examples randomly sampled from the test set. For each example, we randomly shuffle the two response generations, anonymize the model identity, and ask a human annotator to choose which response generation is more topically coherent based on the conversation history. As Table 6 shows, the A-RNN model wins substantially more often than the RNN model.

MovieTriples Ubuntu Troubleshoot
Not distinguishable
RNN wins
A-RNN wins
Table 6: Human Evaluaton: A-RNN vs. RNN

5.5 Qualitative Analysis

Next, we qualitatively evaluate the effectiveness of our A-RNN model through visualizations of the attention and outputs on both datasets. Figure 2 provides a visualization of the attention for a subset of the words in the generation for the two datasets. The last line in both Fig. 1(a) and Fig. 1(b) presents the generated response and we highlight in bold two output words (one on the left and one on the right) for two time steps. For each highlighted generated word, we visualize the attention weights for words in the conversation history (i.e., words in the preceding turns and those previously generated in the output response), where darker shades indicate larger attention weights. As the figure indicates, the attention mechanism helps learn a better RNN language model that promotes topic coherence, by learning to associate the currently-generated word with informative context words in the conversation history. As shown in Figure 3(a), the A-RNN generates meaningful and topically coherent responses on the MovieTriples dataset. In comparison, the vanilla RNN tends to produce generic answers, such as “i don’t know”. Similarly, the A-RNN follows up with useful questions on the Ubuntu Troubleshoot dataset (Fig. 3(b)).

6 Conclusion

We investigate how to improve the performance of a recurrent neural network dialogue model via an attention mechanism, and how to promote topic and saliency aware conversation continuation. Our attention-RNN language model increases the scope of attention continuously as the conversation progresses (which distinguishes it from standard attention with fixed scope in a sequence-to-sequence models) such that each generated word can be associated with its most related words in the conversation history. We evaluate this simple model on two large dialogue datasets (MovieTriples and Ubuntu Troubleshoot), and achieve the best results reported to-date on multiple dialogue metrics (including complementary diversity-based metrics), performing better than gate-based RNN memory models. We also promote topic concentration by adopting LDA-based reranking, further improving performance.

Acknowledgments

We thank Iulian Serban, Yi Luan, and the anonymous reviewers for sharing their datasets and for their helpful discussion. We thank NVIDIA Corporation for donating GPUs used in this research.

Appendix A Appendix

a.1 LDA-based Re-Ranking Details

While the trained attention-RNN dialogue model generates a natural language continuation of a conversation while maintaining topic concentration by token saliency/association, some dialogue-level topic-supervision can help to encourage generations that are more topic-aware. Such supervision is not commonly available, and we use unsupervised methods to learn document-level latent topics, and adopt the learned topic model to select the best continuation based on document-level topic-matching.

We choose Latent Dirichlet Allocation (LDA) [Blei, Ng, and Jordan2003, Blei and Lafferty2009] due to its demonstrated ability to learn a distribution over latent topics given a collection of documents. This generative model assumes that documents arise from multiple topics, each of which is defined as a distribution over a fixed vocabulary of terms. Specifically, we can assume that topics are associated with a collection of dialogues in the dialogue corpus, and that each dialogue exhibits these topics with different likelihoods . Let and denote a - and -dimensional Dirichlet distributions with scalar parameter and vector parameter , respectively. The associated and can be learned by various methods [Blei, Ng, and Jordan2003, Blei and Lafferty2009, Hoffman, Bach, and Blei2010], and the topic proportions of a (possibly unseen) dialogue can be estimated with the learned topic structure . We let denote the estimated topic proportions of any dialogue using the learned topic structure .

We use the training set to learn the topic structure with topics. Having generated a set of continuations for each unseen dialogue during testing, we use the topic structure to compute the topic proportions for the dialogue and its continuations, and , respectively. Next, we employ a pre-defined metric to compute the similarity between the topic proportions for the dialogue and each continuation . Finally, we compute a weighted score , where and is the log-likelihood of the continuation conditioned on the dialogue . The hyper-parameters and are tuned on a held-out development set according to the target evaluation metrics (defined shortly).

Concurrent with our work, luan-16 (luan-16) also propose using learned topic representations of the given conversation as an extra feature in a language model in order to enhance the topic coherence of the generation. They define the function as . As we show in the Results and Analysis section, our attention-RNN language models significantly outperform this approach, though it is possible to further improve the quality of the generation by combining these methods i.e., by defining and also using an LDA-based re-ranker.

a.2 Dataset Details

We train and evaluate the models on two large natural language dialogue datasets, MovieTriples (pre-processed by serban-16 (serban-16)) and Ubuntu Troubleshoot (pre-processed by luan-16 (luan-16)). The dialogue within each of these datasets consists of a sequence of utterances (turns), each of which is a sequence of tokens (words). Table 7 provides the statistics for these two datasets.555Following the approach of luan-16 (luan-16), we randomly sample nine utterances as negative examples of the last utterance for each conversation in the Ubuntu Troubleshoot for the development set.

MovieTriples Ubuntu
training size
dev size
test size
turns / dialogue
avg. tokens / dialogue
vocabulary size
Table 7: Dataset Statistics

a.3 Evaluation Metrics

For the sake of comparison, we closely follow previous work and adopt several standard (and complementary) evaluation metrics (perplexity, word error rate, recall@N and BLEU, diversity-based Distinct-1). On the MovieTriples dataset, we use perplexity (PPL) and word error rate (WER) as is done in previous work, due to the metrics’ ability to appropriately quantify the performance of a language model by measuring its ability to learn the syntactic structure of dialogue. Following serban-16 (serban-16), we adopt two versions for each metric: i) PPL as the word-level perplexity over the entire dialogue conversation; ii) PPL@L as the word-level perplexity over the last utterance of the conversation; iii) WER as the word error rate over the entire dialogue conversation; and iv) WER@L as the word error rate over the last utterance of the conversation. We use all four metrics for model selection on the development set and for evaluation on the test set.

On the Ubuntu Troubleshoot dataset, we follow previous work and use PPL and recall@N to evaluate the performance of the proposed language models. recall@N [Manning et al.2008] evaluates a model by measuring how often the model ranks the correct dialogue continuation within top-N range given candidates. Additionally, we also adopt BLEU score [Papineni et al.2001]

, to evaluate the quality of generations produced by the models. While not perfect, we believe that BLEU score is a suitable metric for evaluating topic-concentration, as it evaluates the generated continuation by counting explicit n-gram match relative to a set of references, and because word-matching can be a reasonable proximity for topic-matching.

Finally, we also present evaluation using the diversity-based Distinct-1 metric (proposed by li-16-diversity (li-16-diversity)) to measure the ability of the A-RNN to promote diversity in the generations, because typical neural dialogue models generate generic, safe responses (technically appropriate but not informative, e.g., “I don t know”). Distinct-1 is computed as the number of distinct unigrams in the generation scaled by the total number of generated tokens.

a.4 Training Details

For the MovieTriples dataset, we follow the same procedure as serban-16 (serban-16) and first pretrain on a large Q-A SubTitle dataset  [Ameixa et al.2014] and then fine-tune on the target MovieTriples dataset. The Q-A SubTitle dataset contains approximately M question-answer pairs, from which we randomly sample pairs as the held-out set. We performed early-stopping according to the PPL score on the held-out set. We tuned the dimensionality of the RNN hidden states, and chose . We similarly use a setting of on the Ubuntu Troubleshoot dataset. We use Adam [Kingma and Ba2015] for optimization in RNN back-propagation.

When training the attention-RNN dialogue model, we perform early-stopping and model selection based on the PPL values on the development set, following luan-16 (luan-16). For the LDA-based re-ranker, we choose the two hyper-parameters, i.e., the number of topics and ranking weight , based on the values of recall@N and BLEU score on the development set. In the end, we choose out of and out of .

a.5 Output Examples

History: [align=center]A: i knew how much you wanted it . R: but i was going to buy it for myself . A-RNN: [align=center,hidealllines=true] A: i ’ d like to buy you a drink . RNN: [align=center,hidealllines=true] A: i don ‘ t know . Reference: [align=center,hidealllines=true] A: i wanted to give it to you . for our anniversary . we met five years ago today . History: [align=center]A: person , but this time i got the money . R: if you give it to me , i might forget i found you . A-RNN: [align=center,hidealllines=true] A: i ’ ll give it to you . RNN: [align=center,hidealllines=true] A: i don ’ t know . Reference: [align=center,hidealllines=true] A: i don ’ t have it with me . tell person .
((a)) MovieTriples
History: [align=center]A: so i need to make an ntfs partition …. help ? R: install ntfsprogs R: it will add the functionality to gparted A-RNN: [align=center,hidealllines=true] R: sudo apt-get install ntfsprogs RNN: [align=center,hidealllines=true] R: how do i do that ? Reference: [align=center,hidealllines=true] R: sudo apt-get update ; sudo apt-get install ntfsprogs History: [align=center]A: i have really choppy streaming video in 9.10 , any way to fix that ? A: any way to fix choppy streaming video in 9.10 ? R: what kind of video A-RNN: [align=center,hidealllines=true] R: what video card do you have ? RNN: [align=center,hidealllines=true] R: what are you trying to do ? Reference: [align=center,hidealllines=true] R: what cpu do you have
((b)) Ubuntu Troubleshoot
Figure 4: Examples of responses generated by the A-RNN along with the reference response.

We qualitatively evaluate the effectiveness of our A-RNN model through visualizations of the output on both datasets. The A-RNN generates meaningful and topically coherent responses on the MovieTriples dataset (Fig. 3(a)). In comparison, the vanilla RNN tends to produce generic answers, such as “i don’t know”. On the Ubuntu Troubleshoot dataset, the A-RNN either provides promising technical solutions or follows up with useful questions (Fig. 3(b)).

References

  • [Ameixa et al.2014] Ameixa, D.; Coheur, L.; Fialho, P.; and Quaresma, P. 2014. Luke, I am your father: dealing with out-of-domain requests by using movies subtitles. In Intelligent Virtual Agents, 13–21. Springer.
  • [Bahdanau, Cho, and Bengio2015] Bahdanau, D.; Cho, K.; and Bengio, Y. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations (ICLR).
  • [Bengio et al.2003] Bengio, Y.; Ducharme, R.; Vincent, P.; and Jauvin, C. 2003. A neural probabilistic language model.

    Journal of Machine Learning Research

    .
  • [Blei and Lafferty2009] Blei, D. M., and Lafferty, J. D. 2009. Topic models. Text mining: Classification, clustering, and applications 10(71):34.
  • [Blei, Ng, and Jordan2003] Blei, D. M.; Ng, A. Y.; and Jordan, M. I. 2003. Latent Dirichlet allocation. the Journal of machine Learning research.
  • [Crow1983] Crow, B. 1983. Topic shifts in couples’ conversations. Conversational coherence: Form, structure, and strategy 136–156.
  • [Dušek and Jurčíček2016] Dušek, O., and Jurčíček, F. 2016.

    A context-aware natural language generator for dialogue systems.

    In Proceedings of SIGDIAL.
  • [Goldberg1983] Goldberg, J. 1983. A move towards describing conversational coherence. Conversational coherence: Form, structure, and strategy 25–45.
  • [Hoffman, Bach, and Blei2010] Hoffman, M.; Bach, F. R.; and Blei, D. M. 2010. Online learning for latent dirichlet allocation. In Advances in Neural Information Processing Systems (NIPS).
  • [Kingma and Ba2015] Kingma, D., and Ba, J. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR).
  • [Li et al.2016a] Li, J.; Galley, M.; Brockett, C.; Gao, J.; and Dolan, B. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
  • [Li et al.2016b] Li, J.; Galley, M.; Brockett, C.; Gao, J.; and Dolan, B. 2016b. A persona-based neural conversation model. arXiv preprint arXiv:1603.06155.
  • [Luan, Ji, and Ostendorf2016] Luan, Y.; Ji, Y.; and Ostendorf, M. 2016. LSTM based conversation models. arXiv preprint arXiv:1603.09457.
  • [Manning et al.2008] Manning, C. D.; Raghavan, P.; Schütze, H.; et al. 2008. Introduction to information retrieval, volume 1. Cambridge university press Cambridge.
  • [Mei, Bansal, and Walter2016a] Mei, H.; Bansal, M.; and Walter, M. R. 2016a. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In

    Proceedings of the National Conference on Artificial Intelligence (AAAI)

    .
  • [Mei, Bansal, and Walter2016b] Mei, H.; Bansal, M.; and Walter, M. R. 2016b. What to talk about and how? Selective generation using LSTMs with coarse-to-fine alignment. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics – Human Language Technologies (NAACL HLT).
  • [Mikolov2010] Mikolov, T. 2010. Recurrent neural network based language model. In Proceedings of Interspeech.
  • [Mnih et al.2014] Mnih, V.; Hees, N.; Graves, A.; and Kavukcuoglu, K. 2014. Recurrent models of visual attention. In Advances in Neural Information Processing Systems (NIPS).
  • [Papineni et al.2001] Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2001. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 311–318.
  • [Parikh et al.2016] Parikh, A. P.; Täckström, O.; Das, D.; and Uszkoreit, J. 2016. A decomposable attention model for natural language inference. arXiv preprint arXiv:1606.01933.
  • [Ritter, Cherry, and Dolan2011] Ritter, A.; Cherry, C.; and Dolan, W. B. 2011. Data-driven response generation in social media. In

    Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)

    .
  • [Sanders1983] Sanders, R. 1983. Tools for cohering discourse and their strategic utilization. Conversational coherence: Form, structure, and strategy 67–80.
  • [Serban et al.2016] Serban, I. V.; Sordoni, A.; Bengio, Y.; Courville, A.; and Pineau, J. 2016. Building end-to-end dialogue systems using generative hierarchical neural networks. In Proceedings of the National Conference on Artificial Intelligence (AAAI).
  • [Shang, Lu, and Li2015] Shang, L.; Lu, Z.; and Li, H. 2015. Neural responding machine for short-text conversation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
  • [Sigman1983] Sigman, S. 1983. Some multiple constraints placed on conversational topics. Conversational coherence: Form, structure, and strategy.
  • [Sordoni et al.2015] Sordoni, A.; Galley, M.; Auli, M.; Brockett, C.; Ji, Y.; Mitchell, M.; Nie, J.-Y.; Gao, J.; and Dolan, B. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
  • [Sutskever, Vinyals, and Lee2014] Sutskever, I.; Vinyals, O.; and Lee, Q. V. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS).
  • [Tran, Bisazza, and Monz2016] Tran, K.; Bisazza, A.; and Monz, C. 2016. Recurrent memory network for language modeling. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics – Human Language Technologies (NAACL HLT).
  • [Vinyals and Le2015] Vinyals, O., and Le, Q. 2015. A neural conversational model. In

    ICML Deep Learning Workshop

    .
  • [Wen et al.2015] Wen, T.-H.; Gašić, M.; Mrkšić, N.; Su, P.-H.; Vandyke, D.; and Young, S. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
  • [Wen et al.2016] Wen, T.-H.; Gašić, M.; Mrkšić, N.; M. Rojas-Barahona, L.; Su, P.-H.; Vandyke, D.; and Young, S. 2016. Multi-domain neural network language generation for spoken dialogue systems. In Proceedings of the Conference on North American Chapter of the Association for Computational Linguistics (NAACL). Association for Computational Linguistics.
  • [Winograd1971] Winograd, T. 1971. Procedures as a representation for data in a computer program for understanding natural language. Technical report, DTIC Document.
  • [Xu et al.2015] Xu, K.; Ba, J.; Kiros, R.; Cho, K.; Courville, A.; Salakhutdinov, R.; Zemel, R.; and Bengio, Y. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning (ICML).
  • [Yao et al.2016] Yao, K.; Peng, B.; Zweig, G.; and Wong, K.-F. 2016. An attentional neural conversation model with improved specificity. arXiv preprint arXiv:1606.01292.