A Neural Network Approach to Context-Sensitive Generation of Conversational Responses

We present a novel response generation system that can be trained end to end on large quantities of unstructured Twitter conversations. A neural network architecture is used to address sparsity issues that arise when integrating contextual information into classic statistical models, allowing the system to take into account previous dialog utterances. Our dynamic-context generative models show consistent gains over both context-sensitive and non-context-sensitive Machine Translation and Information Retrieval baselines.


page 1

page 2

page 3

page 4


Emulating Human Conversations using Convolutional Neural Network-based IR

Conversational agents ("bots") are beginning to be widely used in conver...

AARGH! End-to-end Retrieval-Generation for Task-Oriented Dialog

We introduce AARGH, an end-to-end task-oriented dialog system combining ...

Migratable AI: Personalizing Dialog Conversations with migration context

The migration of conversational AI agents across different embodiments i...

Towards Retrieval-based Conversational Recommendation

Conversational recommender systems have attracted immense attention rece...

HeterMPC: A Heterogeneous Graph Neural Network for Response Generation in Multi-Party Conversations

Recently, various response generation models for two-party conversations...

Exploring the context of recurrent neural network based conversational agents

Conversational agents have begun to rise both in the academic (in terms ...

Neural Contextual Conversation Learning with Labeled Question-Answering Pairs

Neural conversational models tend to produce generic or safe responses i...

1 Introduction

Until recently, the goal of training open-domain conversational systems that emulate human conversation has seemed elusive. However, the vast quantities of conversational exchanges now available on social media websites such as Twitter and Reddit raise the prospect of building data-driven models that can begin to communicate conversationally. The work of Ritter2011, for example, demonstrates that a response generation system can be constructed from Twitter conversations using statistical machine translation techniques, where a status post by a Twitter user is “translated” into a plausible looking response.

However, an approach such as that presented in Ritter2011 does not address the challenge of generating responses that are sensitive to the context of the conversation. Broadly speaking, context may be linguistic or involve grounding in the physical or virtual world, but we here focus on linguistic context. The ability to take into account previous utterances is key to building dialog systems that can keep conversations active and engaging. Figure 1

illustrates a typical Twitter dialog where the contextual information is crucial: the phrase “good luck” is plainly motivated by the reference to “your game” in the first utterance. In the MT model, such contextual sensitivity is difficult to capture; moreover, naive injection of context information would entail unmanageable growth of the phrase table at the cost of increased sparsity, and skew towards rarely-seen context pairs. In most statistical approaches to machine translation, phrase pairs do not share statistical weights regardless of their intrinsic semantic commonality.

Figure 1: Example of three consecutive utterances occurring between two Twitter users and .

We propose to address the challenge of context-sensitive response generation by using continuous representations or embeddings of words and phrases to compactly encode semantic and syntactic similarity. We argue that embedding-based models afford flexibility to model the transitions between consecutive utterances and to capture long-span dependencies in a domain where traditional word and phrase alignment is difficult [Ritter et al.2011]

. To this end, we present two simple, context-sensitive response-generation models utilizing the Recurrent Neural Network Language Model (RLM) architecture of 

[Mikolov et al.2010]. These models first encode past information in a hidden continuous representation, which is then decoded by the RLM to promote plausible responses that are simultaneously fluent and contextually relevant. Unlike typical complex task-oriented multi-modular dialog systems [Young2002, Stent and Bangalore2014], our architecture is completely data-driven and can easily be trained end-to-end using unstructured data without requiring human annotation, scripting, or automatic parsing.

This paper makes the following contributions. We present a neural network architecture for response generation that is both context-sensitive and data-driven. As such, it can be trained from end to end on massive amounts of social media data. To our knowledge, this is the first application of a neural-network model to open-domain response generation, and we believe that the present work will lay groundwork for more complex models to come. We additionally introduce a novel multi-reference extraction technique that shows promise for automated evaluation.

2 Related Work

Our work naturally lies in the path opened by Ritter2011, but we generalize their approach by exploiting information from a larger context. Ritter et al. and our work represent a radical paradigm shift from other work in dialog. More traditional dialog systems typically tease apart dialog management [Young2002] from response generation [Stent and Bangalore2014]

, while our holistic approach can be considered a first attempt to accomplish both tasks jointly. While there are previous uses of machine learning for response generation

[Walker et al.2003], dialog state tracking [Young et al.2010], and user modeling [Georgila et al.2006], many components of typical dialog systems remain hand-coded: in particular, the labels and attributes defining dialog states. In contrast, the dialog state in our neural network model is completely latent and directly optimized towards end-to-end performance. In this sense, we believe the framework of this paper is a significant milestone towards more data-driven and less hand-coded dialog processing.

Continuous representations of words and phrases estimated by neural network models have been applied on a variety of tasks ranging from Information Retrieval (IR) 

[Huang et al.2013, Shen et al.2014], Online Recommendation [Gao et al.2014b], Machine Translation (MT) [Auli et al.2013, Cho et al.2014, Kalchbrenner and Blunsom2013, Sutskever et al.2014], and Language Modeling (LM) [Bengio et al.2003, Collobert and Weston2008]

. Gao et al. Gao2014 successfully use an embedding model to refine the estimation of rare phrase-translation probabilities, which is traditionally affected by sparsity problems. Robustness to sparsity is a crucial property of our method, as it allows us to capture context information while avoiding unmanageable growth of model parameters.

Our work extends the Recurrent Neural Network Language Model (RLM) of [Mikolov et al.2010]

, which uses continuous representations to estimate a probability function over natural language sentences. We propose a set of conditional RLMs where contextual information (i.e., past utterances) is encoded in a continuous context vector to help generate the response. Our models differ from most previous work in the way the context vector is constructed. For example, Mikolov2012 and Auli2013 use a pre-trained topic model. In our models, the context vector is learned along with the conditional RLM that generates the response. Additionally, the learned context encodings do not exclusively capture contentful words. Indeed, even “stop words” can carry discriminative power in this task; for example, all words in the utterance “how are you?” are commonly characterized as stop words, yet this is a contentful dialog utterance.

3 Recurrent Language Model

We give a brief overview of the Recurrent Language Model (RLM) [Mikolov et al.2010] architecture that our models extend. A RLM is a generative model of sentences, i.e., given sentence , it estimates:


The model architecture is parameterized by three weight matrices, : an input matrix , a recurrent matrix and an output matrix , which are usually initialized randomly. The rows of the input matrix contain the -dimensional embeddings for each word in the language vocabulary of size . Let us denote by both the vocabulary token and its one-hot representation, i.e., a zero vector of dimensionality with a corresponding to the index of the token. The embedding for is then obtained by . The recurrent matrix keeps a history of the subsequence that has already been processed. The output matrix projects the hidden state into the output layer , which has an entry for each word in the vocabulary

. This value is used to generate a probability distribution for the next word in the sequence. Specifically, the forward pass proceeds with the following recurrence, for



where is a non-linear function applied element-wise, in our case the logistic sigmoid. The recurrence is seeded by setting

, the zero vector. The probability distribution over the next word given the previous history is obtained by applying the softmax activation function:


The RLM is trained to minimize the negative log-likelihood of the training sentence :


The recurrence is unrolled backwards in time using the back-propagation through time (BPTT) algorithm [Rumelhart et al.1988], and gradients are accumulated over multiple time-steps.

Figure 2: Compact representation of an RLM (left) and unrolled representation for two time steps (right).

4 Context-Sensitive Models

We distinguish three linguistic entities in a conversation between two users and : the context111In this work, the context is purely linguistic, but future work might integrate further contextual information, e.g., geographical location, time information, or other forms of grounding. , the message and response . The context represents a sequence of past dialog exchanges of any length; then emits a message to which reacts by formulating its response (see Figure 1).

We use three context-based generation models to estimate a generation model of the response , , conditioned on past information and :


These three models differ in the manner in which they compose the context-message pair .

4.1 Tripled Language Model

In our first model, dubbed RLMT, we straightforwardly concatenate each utterance , , into a single sentence and train the RLM to minimize . Given and , we compute the probability of the response as follows: we perform the forward propagation over the known utterances and to obtain a hidden state encoding useful information about previous utterances. Subsequently, we compute the likelihood of the response from that hidden state.

An issue with this simple approach is that the concatenated sentence will be very long on average, especially if the context comprises multiple utterances. Modelling such long-range dependencies with an RLM is difficult and is still considered an open problem [Pascanu et al.2013]. We will consider RLMT as an additional context-sensitive baseline for the models we present next.

4.2 Dynamic-Context Generative Model I

The above limitation of RLMT can be addressed by strengthening the context bias. In our second model (DCGM-I), the context and the message are encoded into a fixed-length vector representation the is used by the RLM to decode the response. This is illustrated in Figure 3 (left). First, we consider and as a single sentence and compute a single bag-of-words representation . Then, is provided as input to a multilayered non-linear forward architecture that produces a fixed-length representation that is used to bias the recurrent state of the decoder RLM. At training time, both the context encoder and the RLM decoder are learned so as to minimize the negative log-probability of the generated response.

Figure 3: Compact representations of DCGM-I (left) and DCGM-II (right). The decoder RLM receives a bias from the context encoder. In DCGM-I, we encode the bag-of-words representation of both and in a single vector . In DCGM-II, we concatenate the representations and on the first layer to preserve order information.

The parameters of the model are , where are the weights for the layers of the feed-forward context networks. The fixed-length context vector is obtained by forward propagation of the network:


The rows of contain the embeddings of the vocabulary.222Notice that the first layer of the encoder network is linear. We found that this helps learning the embedding matrix as it reduces the vanishing gradient effect partially due to stacking of squashing non-linearities [Pascanu et al.2013]. These are different from those employed in the RLM and play a crucial role in promoting the specialization of the context encoder to a distinct task. The hidden layer of the decoder RLM takes the following form:


This model conditions on the previous utterances via biasing the hidden layer state on the context representation . Note that the context representation does not change through time. This is useful because: (a) it forces the context encoder to produce a representation general enough to be useful for generating all words in the response and (b) it helps the RLM decoder to remember context information when generating long responses.

4.3 Dynamic-Context Generative Model II

Because DCGM-I does not distinguish between and , that model has the propensity to underestimate the strong dependency that holds between and . Our third model (DCGM-II) addresses this issue by concatenating the two linear mappings of the bag-of-words representations and in the input layer of the feed-forward network representing and (see Figure 3 right). Concatenating continuous representations prior to deep architectures is a common strategy to obtain order-sensitive representations [Bengio et al.2003, Devlin et al.2014].

The forward equations for the context encoder are:


where denotes the concatenation of and vectors. In DCGM-II, the bias on the recurrent hidden state and the probability distribution over the next token are computed as described in Eq. 7.

5 Experimental Setting

5.1 Dataset Construction

For computational efficiency and to alleviate the burden of human evaluators, we restrict the context sequence to a single sentence. Hence, our dataset is composed of “triples” consisting of three sentences. We mined 127M context-message-response triples from the Twitter FireHose, covering the 3-month period June 2012 through August 2012. Only those triples where context and response were generated by the same user were extracted. To minimize noise, we selected triples that contained at least one frequent bigram that appeared more than 3 times in the corpus. This produced a corpus of 29M Twitter triples. Additionally, we hired crowdsourced raters to evaluate approximately 33K candidate triples. Judgments on a 5-point scale were obtained from 3 raters apiece. This yielded a set of 4232 triples with a mean score of 4 or better that was then randomly binned into a tuning set of 2118 triples and a test set of 2114 triples333The Twitter ids of the tuning and test sets along with the code for the neural network models may be obtained from http://research.microsoft.com/convo/. The mean length of responses in these sets was approximately 11.5 tokens, after cleanup (e.g., stripping of emoticons), including punctuation.

5.2 Automatic Evaluation

We evaluate all systems using BLEU [Papineni et al.2002] and METEOR [Banerjee and Lavie2005], and supplement these results with more targeted human pairwise comparisons in Section 6.3. A major challenge in using these automated metrics for response generation is that the set of reasonable responses in our task is potentially vast and extremely diverse. The dataset construction method just described yields only a single reference for each status. Accordingly, we extend the set of references using an IR approach to mine potential responses, after which we have human judges rate their appropriateness. As we see in Section 6.3, it turns out that by optimizing systems towards BLEU using mined multi-references, BLEU rankings align well with human judgments. This lays groundwork for interesting future correlation studies.

Corpus # Triples Avg # Ref [Min,Max] # Ref
Tuning 2118 3.22 [1, 10]
Test 2114 3.58 [1, 10]
Table 1: Number of triples, average, minimum and maximum number of references for tuning and test corpora.

Multi-reference extraction

We use the following algorithm to better cover the space of reasonable responses. Given a test triple , our goal is to mine other responses that fit the context and message pair . To this end, we first select a set of 15 candidate triples using an IR system. The IR system is calibrated in order to select candidate triples for which both the message and the response are similar to the original message and response . Formally, the score of a candidate triple is:


where is the bag-of-words BM25 similarity function [Robertson et al.1995], controls the impact of the similarity between the responses and is a smoothing factor that avoids zero scores for candidate responses that do not share any words with the reference response. We found that this simple formula provided references that were both diverse and plausible. Given a set of candidate triples , human evaluators are asked to rate the quality of the response within the new triples . After human evaluation, we retain the references for which the score is 4 or better on a 5 point scale, resulting in 3.58 references per example on average (Table 1). The average lengths for the responses in the multi-reference tuning and test sets are 8.75 and 8.13 tokens respectively.

5.3 Feature Sets

The response generation systems evaluated in this paper are parameterized as log-linear models in a framework typical of statistical machine translation [Och and Ney2004]. These log-linear models comprise the following feature sets:


MT features are derived from a large response generation system built along the lines of Ritter2011, which is based on a phrase-based MT decoder similar to Moses [Koehn et al.2007]. Our MT feature set includes the following features that are common in Moses: forward and backward maximum likelihood “translation” probabilities, word and phrase penalties, linear distortion, and a modified Kneser-Ney language model [Kneser and Ney1995] trained on Twitter responses. For the translation probabilities, we built a very large phrase table of 160.7 million entries by first filtering out Twitterisms (e.g., long sequences of vowels, hashtags), and then selecting candidate phrase pairs using Fisher’s exact test [Ritter et al.2011]. We also included MT decoder features specifically motivated by the response generation task: Jaccard distance between source and target phrase, Fisher’s exact probability, and a score relating the lengths of source and target phrases.


We also use an IR feature built from an index of triples, whose implementation roughly matches the IR approach described in Ritter2011: For a test triple , we choose as the candidate response iff .


Neither MT nor IR traditionally take into account contextual information. Therefore, we take into consideration context and message matches (CMM), i.e., exact matches between , and . We define features as the [1-4]-gram matches between and the candidate reply and the [1-4]-gram matches between and the candidate reply . These exact matches help capture and promote contextual information in the replies.

Rlmt, Dcgm-I, Dcgm-Ii

We consider the RLM trained on the concatenated triples, denoted as RLMT (Section 4.1), to be a context-sensitive RLM baseline. Each neural network model contributes an additional feature corresponding to the likelihood of the candidate response given context and message.

System BLEU
random 0.33
mt 3.21
human 6.08
Table 2: Multi-reference corpus-level BLEU obtained by leaving one reference out at random.
MT -best BLEU () METEOR () MT 3.60 (-9.5%) 9.19 (-0.9%) CMM 3.33 (-16%) 9.34 (+0.7%) MT + CMM 3.98 (-) 9.28 (-) RLMT 4.13 (+3.7%) 9.54 (+2.7%) DCGM-I 4.26 (+7.0%) 9.55 (+2.9%) DCGM-II 4.11 (+3.3%) 9.45 (+1.8%) DCGM-I + CMM 4.44 (+11%) 9.60 (+3.5%) DCGM-II + CMM 4.38 (+10%) 9.62 (+3.5%) IR -best BLEU () METEOR () IR 1.51 (-55%) 6.25 (-22%) CMM 3.39 (-0.6%) 8.20 (+0.6%) IR + CMM 3.41 (-) 8.04 (-) RLMT 2.85 (-16%) 7.38 (-8.2%) DCGM-I 3.36 (-1.5%) 7.84 (-2.5%) DCGM-II 3.37 (-1.1%) 8.22 (+2.3%) DCGM-I + CMM 4.07 (+19%) 8.67 (+7.8%) DCGM-II + CMM 4.24 (+24%) 8.61 (+7.1%)
Table 3: Context-sensitive ranking results on both MT (left) and IR (right) -best lists, . The subscript indicates the number of features of the models. The log-linear weights are estimated by running one iteration of MERT. We mark by (%) the relative improvements with respect to the reference system ().

5.4 Model Training

The proposed models are trained on a 4M subset of the triple data. The vocabulary consists of the most frequent

words. In order to speed up training, we use the Noise-Contrastive Estimation (NCE) loss, which avoids repeated summations over

by approximating the probability of the target word [Gutmann and Hyvärinen2010]. Parameter optimization is done using Adagrad [Duchi et al.2011] with a mini-batch size of 100 and a learning rate , which we found to work well on held-out data. In order to stabilize learning, we clip the gradients to a fixed range

, as suggested in Mikolov2010. All the parameters of the neural models are sampled from a normal distribution

while the recurrent weight

is initialized as a random orthogonal matrix and scaled by 0.01. To prevent over-fitting, we evaluate performance on a held-out set during training and stop when the objective increases. The size of the RLM hidden layer is set to

, where the context encoder is a , , multilayer network. The bottleneck in the middle compresses context information that leads to similar responses and thus achieves better generalization. The last layer embeds the context vector into the hidden space of the decoder RLM.

5.5 Rescoring Setup

We evaluate the proposed models by rescoring the -best candidate responses obtained using the MT phrase-based decoder and the IR system. In contrast to MT, the candidate responses provided by IR have been created by humans and are less affected by fluency issues. The different -best lists will provide a comprehensive testbed for our experiments. First, we augment the -best list of the tuning set with the scores of the model of interest. Then, we run an iteration of MERT [Och2003] to estimate the log-linear weights of the new features. At test time, we rescore the test -best list with the new weights.

6 Results

6.1 Lower and Upper Bounds

Table 2 shows the expected upper and lower bounds for this task as suggested by BLEU scores for human responses and a random response baseline. The random system comprises responses randomly extracted from the triples corpus. human is computed by choosing one reference amongst the multi-reference set for each context-status pair.444For the human score, we compute corpus-level BLEU with a sampling scheme that randomly leaves out one reference - the human sentence to score - for each reference set. This sampling scheme (repeated with 100 trials) is also applied for the MT and random system so as to make BLEU scores comparable. Although the scores are lower than those usually reported in SMT tasks, the ranking of the three systems is unambiguous.


The results of automatic evaluation using BLEU and METEOR are presented in Table 3, where some broad patterns emerge. First, both metrics indicate that a phrase-based MT decoder outperforms a purely IR approach. Second, adding CMM features to the baseline systems helps. Third, the neural network models contribute measurably to improvement: RLMT and DCGM models outperform baselines, and DCGM models provide more consistent gains than RLMT.

MT vs. IR

BLEU and METEOR scores indicate that the phrase-based MT decoder outperforms a purely IR approach, despite the fact that IR proposes fluent human generated responses. This may be because the IR model only loosely captures important patterns between message and response: It ranks candidate responses solely by the similarity of their message with the message of the test triple (§5.3). As a result, the top ranked response is likely to drift from the purpose of the original conversation. The MT approach, by contrast, more directly models statistical patterns between message and response.


MT+CMM, totaling 17 features (9 from MT + 8 CMM), improves 0.38 BLEU points, a relative improvement, over the baseline MT model. IR+CMM, with 10 features (IR + word penalty + 8 CMM), benefits even more, attaining 1.8 BLEU points and 1.5 METEOR points over the IR baseline. Figure 4 (a) and (b) plots the magnitude of the learned CMM feature weights for MT+CMM and IR+CMM. CMM features help in both these hypothesis spaces and especially on the IR -best list. Figure 4 (b) supports the hypothesis formulated in the previous paragraph: Since IR solely captures inter-message similarities, the matches between message and response are important, while context matches help in providing additional gains. The phrase-based statistical patterns captured by the MT system do a good job in explaining away 1-gram and 2-gram message matches (Figure 4 (a)) and the performance gain mainly comes from context matches. On the other hand, we observe that 4-gram matches may be important in selecting appropriate responses. Inspection of the tuning set reveals instances where responses contain long subsequences of their corresponding messages, e.g., = “good night best friend, I love you”,

= “I love you too, good night best friend”. Although infrequent, such higher-order n-gram matches, when they occur, may provide a more robust signal of the quality of the response than 1- and 2-gram matches, given the highly conversational nature of our dataset.

Figure 4: Comparison of the weights of learned CMM features for MT+CMM and IR+CMM systems (a) et (b) and DCGM-II+CMM on MT and IR (c) and (d).


Both RLMT and DCGM models outperform their respective MT and IR baselines. Both models also exhibit similar performance and show improvements over the MT+CMM models, albeit using a lower dimensional feature space. We believe that their similar performance is due to the limited diversity of MT -best list together with gains in fluency stemming from the strong language model provided by the RLM. In the case of IR models, on the other hand, there is more headroom for improvement and fluency is already guaranteed. Any gains must come from context and message matches. Hence, RLMT underperforms with respect to both DCGM and IR+CMM. The DCGM models appear to have better capacity to retain contextual information and thus achieve similar performance to IR+CMM despite their lack of exact n-gram match information.

In the present experimental setting, no striking performance difference can be observed between the two versions of the DCGM architecture. If multiple sequences were used as context, we expect that the DCGM-II model would likely benefit more owing to the separate encoding of message and context.

System A System B Gain (%) CI
HUMAN MT+CMM 13.6* [12.4,14.8]
DCGM-II MT 1.9* [0.8, 2.9]
DCGM-II+CMM MT 3.1* [2.0, 4.3]
DCGM-II+CMM MT+CMM 1.5* [0.5, 2.5]
DCGM-II IR 5.2* [4.0, 6.4]
DCGM-II+CMM IR 5.3* [4.1, 6.6]
DCGM-II+CMM IR+CMM 2.3* [1.2, 3.4]
Table 4: Pairwise human evaluation scores between System A and B. The first (second) set of results refer to the MT (IR) hypothesis list. The asterisk means agreement between human preference and BLEU rankings.


We also investigated whether mixing exact CMM n-gram overlap with semantic information encoded by the DCGM models can bring additional gains. DCGM-{I-II}+CMM systems each totaling 10 features show increases of up to 0.48 BLEU points over MT+CMM and up to 0.88 BLEU over the model based on Ritter2011. METEOR improvements similarly align with BLEU improvements both for MT and IR lists. We take this as evidence that CMM exact matches and DCGM semantic matches interact positively, a finding that comports with Gao2014, who show that semantic relationships mined through phrase embeddings correlate positively with classic co-occurrence-based estimations. Analysis of CMM feature weights in Figure 4 (c) and (d) suggests that 1-gram matches are explained away by the DCGM model, but that higher order matches are important. It appears that DCGM models might be improved by preserving word-order information in context and message encodings.

6.3 Human Evaluation

Human evaluation was conducted using crowdsourced annotators. Annotators were asked to compare the quality of system output responses pairwise (“Which is better?”) in relation to the context and message strings in the 2114 item test set. Identical strings were held out, so that the annotators only saw those outputs that differed. Paired responses were presented in random order to the annotators, and each pair of responses was judged by 5 annotators.

Table 4 summarizes the results of human evaluation, giving the difference in mean scores (pairwise preference margin) between systems and 95% confidence intervals generated using Welch’s t-test. Identical strings not shown to raters are incorporated with an automatically assigned score of 0.5. The pattern in these results is clear and consistent: context-sensitive systems (+CMM) outperform non-context-sensitive systems, with preference gains as high as approximately 5.3% in the case of DCGM-II+CMM versus IR, and about 3.1% in the case of DCGM-II+CMM versus MT. Similarly, context-sensitive DCGM systems outperform non-DCGM context-sensitive systems by 1.5% (MT) and 2.3% (IR). These results are consistent with the automated BLEU rankings and confirm that our best performing DCGM models outperform both raw baseline and the context-sensitive baseline using CMM features.

6.4 Discussion

# Context Message Generated Response
1 noone can help this time its just i need a support system well im here to support you. whenever you need me. thank you, i appreciate it thank you
2 nothin much, and how’s the book?! its good but i’m only like halfway through cuz i don’t feel like reading. i’m so bored … that’s good! i have the book but i’m bored too.
3 i know! what have you been up to? nothing really. just enjoying summer and working. what about you? that’s good! nothing much just listening to music and you?
4 thanks, your way better though i’m just good at expressing it that’s but trust me you are freakin amazing and waay better i know, i don’t trust you, i love you too! thank you
5 thanks. yeah i am good thanks! good, but angry! sorry you’re angry, been there, i hope it dies down. i hope so too, i’m not angry!!
6 figures. bass. never ate bass. bass is extremely mild with very few bones. my favorite after catfish that’s one of the bones, chuck bass is mine too.
7 today i will walk for miles. clear sky clear mind. love walking. it’s too hot to walk this month. i haven’t moved. frozen. heat freeze. it’s too hot for you to get up to honey if you have frozen yogurt to you.
8 arresting people over “nasty tweets”? please. i’m not condoning what the lad did but arresting him is a tad silly. it appears he may have sent death threats to others while all this was kicking off too. i did! he was armed with nuclear threats? that’s what’s happening to you.
Table 5: Sample responses produced by the MT-based DCGM-II+CMM system.

Table 5

provides examples of responses generated on the tuning corpus by the MT-based DCGM-II+CMM system, our best system in terms of both BLEU and human evaluation. Responses from this system are on average shorter (8.95 tokens) than the original human responses in the tuning set (11.5 tokens). Overall, the outputs tend to be generic or commonplace, but are often reasonably plausible in the context as in examples 1-3, especially where context and message contain common conversational elements. Example 2 illustrates the impact of context-sensitivity: the word “book” in the response is not found in the message. Nonetheless, longer generated responses are apt to degrade both syntactically and in terms of content. We notice that longer responses are likely to present information that conflicts either internally within the response itself, or is at odds with the context, as in examples 4-5. This is not unsurprising, since our model lacks mechanisms both for reflecting agent intent in the response and for maintaining consistency with respect to sentiment polarity. Longer context and message components may also result in responses that wander off-topic or lapse into incoherence as in 6-8, especially when relatively low frequency unigrams (“bass”, “threat”) are echoed in the response. In general, we expect that larger datasets and incorporation of more extensive contexts into the model will help yield more coherent results in these cases. Consistent representation of agent intent is outside the scope of this work, but will likely remain a significant challenge.

7 Conclusion

We have formulated a neural network architecture for data-driven response generation trained from social media conversations, in which generation of responses is conditioned on past dialog utterances that provide contextual information. We have proposed a novel multi-reference extraction technique allowing for robust automated evaluation using standard SMT metrics such as BLEU and METEOR. Our context-sensitive models consistently outperform both context-independent and context-sensitive baselines by up to 11% relative improvement in BLEU in the MT setting and 24% in the IR setting, albeit using a minimal number of features. As our models are completely data-driven and self-contained, they hold the potential to improve fluency and contextual relevance in other types of dialog systems.

Our work suggests several directions for future research. We anticipate that there is much room for improvement if we employ more complex neural network models that take into account word order within the message and context utterances. Direct generation from neural network models is an interesting and potentially promising next step. Future progress in this area will also greatly benefit from thorough study of automated evaluation metrics.


We thank Alan Ritter, Ray Mooney, Chris Quirk, Lucy Vanderwende, Susan Hendrich and Mouni Reddy for helpful discussions, as well as the three anonymous reviewers for their comments.


  • [Auli et al.2013] Michael Auli, Michel Galley, Chris Quirk, and Geoffrey Zweig. 2013. Joint language and translation modeling with recurrent neural networks. In Proc. of EMNLP, pages 1044–1054.
  • [Banerjee and Lavie2005] Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proc. of ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Jun.
  • [Bengio et al.2003] Yoshua Bengio, Rejean Ducharme, and Pascal Vincent. 2003. A neural probabilistic language model. Journ. Mach. Learn. Res., 3:1137–1155.
  • [Cho et al.2014] Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. Proc. of EMNLP.
  • [Collobert and Weston2008] Ronan Collobert and Jason Weston. 2008.

    A unified architecture for natural language processing: Deep neural networks with multitask learning.

    In Proc. of ICML, pages 160–167. ACM.
  • [Devlin et al.2014] Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Proc. of ACL.
  • [Duchi et al.2011] John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journ. Mach. Learn. Res., 12:2121–2159.
  • [Gao et al.2014a] Jianfeng Gao, Xiaodong He, Wen tau Yih, and Li Deng. 2014a. Learning continuous phrase representations for translation modeling. In Proc. of ACL, pages 699–709.
  • [Gao et al.2014b] Jianfeng Gao, Patrick Pantel, Michael Gamon, Xiaodong He, and Li Deng. 2014b. Modeling interestingness with deep neural networks. In Proc. of EMNLP, pages 2–13.
  • [Georgila et al.2006] Kallirroi Georgila, James Henderson, and Oliver Lemon. 2006. User simulation for spoken dialogue systems: Learning and evaluation. In Proc. of Interspeech/ICSLP.
  • [Gutmann and Hyvärinen2010] Michael Gutmann and Aapo Hyvärinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proc. of AISTATS, pages 297–304.
  • [Huang et al.2013] Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proc. of CIKM, pages 2333–2338.
  • [Kalchbrenner and Blunsom2013] Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. Proc. of EMNLP, pages 1700–1709.
  • [Kneser and Ney1995] Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for M-gram language modeling. In Proc. of ICASSP, pages 181–184, May.
  • [Koehn et al.2007] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proc. of ACL Demo and Poster Sessions, pages 177–180.
  • [Mikolov and Zweig2012] Tomas Mikolov and Geoffrey Zweig. 2012. Context Dependent Recurrent Neural Network Language Model.
  • [Mikolov et al.2010] Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proc. of INTERSPEECH, pages 1045–1048.
  • [Och and Ney2004] Franz Josef Och and Hermann Ney. 2004. The alignment template approach to machine translation. Comput. Linguist., 30(4):417–449.
  • [Och2003] Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proc. of ACL, pages 160–167.
  • [Papineni et al.2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. of ACL, pages 311–318.
  • [Pascanu et al.2013] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. Proc. of ICML, pages 1310–1318.
  • [Ritter et al.2011] Alan Ritter, Colin Cherry, and William B. Dolan. 2011. Data-driven response generation in social media. In Proc. of EMNLP, pages 583–593.
  • [Robertson et al.1995] Stephen E Robertson, Steve Walker, Susan Jones, et al. 1995. Okapi at TREC-3. In TREC.
  • [Rumelhart et al.1988] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1988. Learning representations by back-propagating errors. In James A. Anderson and Edward Rosenfeld, editors, Neurocomputing: Foundations of Research, pages 696–699. MIT Press, Cambridge, MA, USA.
  • [Shen et al.2014] Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014. A latent semantic model with convolutional-pooling structure for information retrieval. In Proc. of CIKM, pages 101–110.
  • [Stent and Bangalore2014] Amanda Stent and Srinivas Bangalore. 2014. Natural Language Generation in Interactive Systems. Cambridge University Press.
  • [Sutskever et al.2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. V Le. 2014. Sequence to sequence learning with neural networks. Proc. of NIPS.
  • [Walker et al.2003] Marilyn A. Walker, Rashmi Prasad, and Amanda Stent. 2003. A trainable generator for recommendations in multimodal dialog. In Proc. of EUROSPEECH.
  • [Young et al.2010] Steve Young, Milica Gašić, Simon Keizer, François Mairesse, Jost Schatzmann, Blaise Thomson, and Kai Yu. 2010. The hidden information state model: A practical framework for pomdp-based spoken dialogue management. Comput. Speech Lang., 24(2):150–174.
  • [Young2002] Steve Young. 2002. Talking to machines (statistically speaking). In Proc. of INTERSPEECH.