A retrieval-based dialogue system utilizing utterance and context embeddings

10/16/2017 ∙ by Alexander Bartl, et al. ∙ Maastricht University 0

Finding semantically rich and computer-understandable representations for textual dialogues, utterances and words is crucial for dialogue systems (or conversational agents), as their performance mostly depends on understanding the context of conversations. Recent research aims at finding distributed vector representations (embeddings) for words, such that semantically similar words are relatively close within the vector-space. Encoding the "meaning" of text into vectors is a current trend, and text can range from words, phrases and documents to actual human-to-human conversations. In recent research approaches, responses have been generated utilizing a decoder architecture, given the vector representation of the current conversation. In this paper, the utilization of embeddings for answer retrieval is explored by using Locality-Sensitive Hashing Forest (LSH Forest), an Approximate Nearest Neighbor (ANN) model, to find similar conversations in a corpus and rank possible candidates. Experimental results on the well-known Ubuntu Corpus (in English) and a customer service chat dataset (in Dutch) show that, in combination with a candidate selection method, retrieval-based approaches outperform generative ones and reveal promising future research directions towards the usability of such a system.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Text-only based Dialogue systems, also called Conversational Agents, Chatbots or Chatterbots, have become very popular in the research community and for large companies. The reason for the rise in popularity lies in the fact that their ability to interact intelligently with humans has improved significantly due to advancements in hardware technologies and Artificial Intelligence.

One of the latest effective approaches [1]

is to represent words, phrases, or even complete dialogues as fixed-length vectors of floating point numbers, also called embeddings (or distributed representations or feature vectors). The Hierarchical Recurrent Encoder-Decoder (HRED)

[2] and its successors [3], (as well as similar related models) are specifically designed to encode the meaning of textual conversations regarding the special structure that originates from multiple turn-taking speakers.

In our approach, a context embedding, a vector encoding the meaning of a conversation up to a certain time step , encoded by the HRED model, serves as input to the decoder component to generate a textual answer. We explore the performance of a retrieval-based model that uses the utterance- and context-embeddings, previously generated by the HRED model, to find similar conversations and rank possible candidate answers. We argue that a retrieval-based approach, based on embeddings, can outperform the generative approach, as the retrieval of similar conversations is less dependent on high quality embeddings and less susceptible to poorly trained embeddings.

The rest of the paper is organized as follows: We first give an outline of the research around dialogue systems, showing the development from purely script-based systems to deep networks generating answers end-to-end. The proposed pipeline is discussed in Section III

. Experimental setup (datasets, evaluation metrics, models implemented and compared) as well as the results are discussed in Section

IV. Finally, we conclude the paper by summarizing the main findings and outline future work.

Ii Related Work

The purpose of Dialogue Systems (DS), often also termed Conversational Agents (CA) or Chatterbots, is to converse with humans to provide information, help in decision making, perform administrative services, or just for the sake of entertainment [4]. The traditional design of Dialogue Systems [5] follows a modular approach, splitting the system usually into a Natural Language Understanding (NLU) module, a Dialogue Manager and a Natural Language Generation (NLG) unit. The NLU module processes the raw user input and extracts useful information and features that can be used by the Dialogue Manager to update internal states, send queries to a knowledge base or, more generally, find actions based on a script. The NLG acts inversely to the NLU module, receiving features and information from the Dialogue Manager to generate a response that, finally, will be presented to the user.

One of the simplest design approaches for an NLU is to simply spot certain key-words or combinations of them. This is often the general procedure of script-based chatbots and the approach followed by ELIZA. However, there is a long history of attempts [6]

to improve NLU and find better representations of text. With advances in machine learning, the development ranges from statistical modelling of language

[7], semantic parsing [8], skip-gram models [1], and others, to approaches utilizing deep neural architectures [9, 10]

. Neural networks have also been used to improve NLG

[11, 12]

. Recently, Dialogue Managers have made similar advances towards automated solutions, with a focus on reinforcement learning

[13, 14], generating policies of how to interact with humans, based on some state representation.

With the rise of Deep Learning (DL) in recent years [15]

and an increasing company interest in chatterbots, end-to-end Dialogue Systems, such as deep Recurrent Neural Networks, constituting all modules in one model

[2], have become one of the major research topics for Dialogue Systems. The tasks of NLU, NLG, and the Dialogue Manager are performed by a single deep network that is trained to reproduce conversations from a large dataset. Such a system would generate an answer end-to-end from raw user input. Even though training deep RNNs can be considerably difficult [16], only one model would need to be optimized, and one could benefit from the neural model’s capability to generate natural responses [17, 11].

Our proposed pipeline is a combination of a generative- and retrieval-based approach. An encoder model, such as the HRED model (or could be one of its more advanced variations) is trained end-to-end on a textual corpus, using an objective function that is based on how capable the model is of generating the answers in the training set. After the training however, the decoder component of the HRED model is not used to generate answers. Instead, we argue that a retrieval-based approach taking over the NLG part performs equally or better in both general and specific domains.

Iii Model description

Fig. 1: A view of the pipeline implementing the proposed approach. An HRED encoder is used to generate context and response embeddings and an ANN model builds on previous steps to retrieve similar conversations. Finally, the best candidate is selected according to answer- and context-relevance.

The proposed model can be split up into three individual components. The first component, the encoder, utilizes the HRED model to encode raw conversations into embeddings containing the actual meaning. The second component, a retrieval-based approach using an Approximate Nearest Neighbor (ANN) model, is responsible for retrieving similar conversations from a database of embedding- and raw-text-tuples. Given the context of an unfinished conversation, suitable responses are considered to be contained in similar conversations, retrieved by the ANN model. The last model component receives a retrieved set and ranks possible answers based on answer- and context-relevance. The entire pipeline can be seen in Figure 1.

Iii-a Gated Recurrent Unit (GRU)

Recurrent Neural Networks (RNNs) have been designed to process sequential data by encoding historic information into a hidden state. When optimizing the network to predict future values using the current input and historic values, the hidden state naturally becomes a new representation form of the processed data. LSTMs, first introduced 1997, are an attempt to increase a RNN’s capabilities to remember long-term dependencies by replacing hidden units with more complex memory cells, capable of controlling the information flow in and out of the cells.

The more recently proposed GRUs are similar to LSTM cells aiming to improve a RNN’s capabilities to remember long-term dependencies. However, they use a different gate design, have fewer parameters to train and come without an additional cell state. Two gates, the reset and update gates and , operate directly on the hidden state, i.e., the hidden layer. Parametrized by , and , while conditioned on the current input and previous result , GRU gate vectors are computed as:



being the sigmoid function. The update gate

combines the function of the input and forget gate by controlling how much the new hidden state (

) is defined by either the current input or the last hidden state, using linear interpolation:


with being the candidate activation. The reset gate is used to calculate , controlling similarly how much of the previous hidden state to keep:


with being the hyperbolic tangent function.

Iii-B Hierarchical Recurrent Encoder-Decoder

The HRED model essentially consists of three stacked RNNs: the utterance encoder, context encoder, and utterance decoder, each of them depending on the result of its predecessor and operating on distributed representations.

Formally, a dialogue , the input to such a model, can be represented as a sequence of utterances , with being a sequence of word indices , each of them usually pointing to a vocabulary reference or directly to a word embedding. These become the input to the utterance encoder.

Iii-B1 Encoding steps

To better capture long-term dependencies, the GRU gating function is used for the individual RNNs. For a simplified notation, the GRU can be expressed compactly by combining equations 1, 2 and 3:


computing current hidden state , conditioned on the previous, and on current input . To comply with the HRED notation, the utterance embedding of the current utterance , including word , is calculated as:


Applying equation 5 consecutively on word embeddings , results in an equally-sized set of hidden states , where the last hidden state is the summary of all words in the same utterance. As such, we denote to be the hidden state that represents utterance .

Using this encoding approach, a set of utterances is encoded into hidden states . Those are used as input to the GRU-based context encoder, similar to how word embeddings acted as input to the utterance encoder. As such, context embeddings are a summary of utterances and represent entire dialogues. They are computed as:


Iii-B2 Decoding step

In addition to encoding a sequence of embeddings into a hidden state, the decoder component generates word probabilities over a vocabulary, given some context

and previous words .

Firstly, to condition the decoder RNN on previous utterances, the initialization of its hidden state is based on the context encoders last hidden state . If not designed explicitly, context and decoder RNN usually have different hidden state dimensionalities, which is why an additional network layer is added to project context embeddings into the decoder space:


with parameters and and being the decoder RNN’s initial hidden state.

Given a set of words , having been previously generated or representing a training example, the decoder RNN hidden state is similarly computed as it was done for the encoder RNNs:


processing words consecutively. The first iteration uses the hidden state computed by equation 7 and a zero-value embedding for to predict the first word of an utterance.

Using both, the hidden state and word embedding of , the word embedding of current word is then predicted as:


with the additional parameters , and . and control which part of the previous context- and word embedding contribute to the new word embedding and how much of that part is used.

By calculating the dot product of such generated word embeddings with the embeddings in a vocabulary, one can compute the similarity between prediction and existing words, with the most similar word being the most likely one. The actual probability of a word occurring next is based on this similarity:


with being the word embedding of word , the predicted word embedding and the vocabulary size.

By computing probabilities for each word in the vocabulary, one can create a distribution from which words can be sampled. Pushing sampled words back into the decoder allows to generate the next word, extending an utterance until an end-of-sequence meta token has been reached. Possible generations can be explored using probability based search techniques such as Beam Search [18].

Iii-C Retrieval Model

Using the encoded corpus as a database of vectors, a Nearest Neighbor Search (NNS) algorithm can be used to find close embeddings in the whole set. For this purpose, an ANN approach has been considered, as general space-partitioning approaches, aiming to improve the NNS performance, suffer from the curse of dimensionality


Locality Sensitive Hashing (LSH) [20, 21] is an ANN approach that uses a set of hashing functions to project similar data points into buckets and as such, significantly restricts the search space to the size of the bucket. For a projection, a binary string label is constructed by applying different hashing functions to a single data point, where the output of such a function is either one or zero. The desired goal of a hashing function is to output the same label for similar data points and differing labels for dissimilar ones. Therefore, binary string labels that are similar, indicate that also the original data points are similar. The string label is then used as a key to index a bucket of similar data points, where a brute force approach can be applied on a much smaller set. A collection of buckets is called a hash-table and tables constitute the entire model.

One of the main issues with the basic LSH algorithm [21] is that choosing the optimal number of hashing functions and number of tables requires one to know the most suitable value for , the threshold separating similar and dissimilar points. The LSH-Forest algorithm solves this issue by allowing labels with variable length and thus, eliminating parameter .

Instead of linking fixed-length labels to buckets, the label string is stored in a prefix tree, a binary tree (also called ’trie’), in which keys are not contained within nodes but derived from the path that leads from the root to a node.

For the case that two points are very similar or equal, the length-limiting parameter prevents their labels from growing too large.

Each level of the tree is associated with a different hashing function, sampled uniformly and with replacement from a family of hashing functions . Such a tree, an LSH-Tree, is the equivalent to an LSH-based hash-table and the composition of trees is an LSH-Forest.

Given a query point , finding close neighbors in a set of LSH-Trees is performed in two phases. First, in a top-down phase or descent, each tree is searched for the leaf node with the best match to the binary string label of . Labels are computed individually for each tree, starting from the root and extending the label until a leaf node is reached.

Inspecting the matches from all trees, the match with the longest prefix defines the tree-level from which the bottom-up accumulation, the second phase, begins.

Iii-D Candidate Selection

Given a query context of an unfinished conversation, using the previously discussed LSH-Forest algorithm, one can retrieve a candidate set from a database of encoded conversations. Candidate answers will be scored based on the matching degree between the retrieved and the original context in terms of question-to-question similarity or answer relevance or other text-based features. However, the scoring functions introduced in this section will solely be based on vector comparison metrics, such as the cosine similarity, as text-based comparison is less rewarding and more difficult and tedious to implement. For the sake of clarity, the query context embedding is defined as

, the textual candidates as , the context embeddings of candidates as , and the utterance embeddings of candidates as .

Iii-D1 Context Relevance

The similarity of two conversations or the distance between a query context and a candidate context has, intuitively, a big impact on the retrieved answer, i.e, the more two questions are similar, the higher the probability that the answers are similar as well. If the cosine similarity has been chosen as the distance function , the labels returned by the nearest neighbor search are already sorted by context-to-context distance. Formally, given a candidate response and a query context , the Context Relevance (CR) cost function is defined as:

Fig. 2: A simple illustration of how CR is computed for a single candidate.

Iii-D2 Answer Relevance

By manual inspection of near neighbors, it became apparent that the correct answer is usually represented or almost captured in many topic-related candidates. Assuming that the most suitable topic for answering is dominantly represented amongst candidates, responses are ranked based on how much they capture the general topic. Formally, the cost of a response is defined by the accumulated similarity between its respective embedding and the utterance embeddings of all other candidates (See Figure 3), normalized by length :

Fig. 3: An image showing how the AR cost of a single candidate is accumulated by computing the cosine similarity with other candidate’s utterance embeddings.

Iii-D3 Combining Context and Answer Relevance

The problem with the previous approach is that the candidates that are off-topic still contribute to the answer relevance cost. Therefore, in a pre-step, according to the previously described context relevance metric, the top candidates are accumulated to represent the best general answer topic. In the next step, candidates are ranked based on their similarity to these responses. Formally, combined Context and Answer Relevance (CAR) is defined as:


with .

Iv Experiments

Iv-a Datasets

The first dataset we use is the Ubuntu Dialogue Corpus which has been studied in most state of the art systems (similar to HRED). The ubuntu dataset contains almost 1 million multi-turn dialogues, with a total of over 7 million utterances and 100 million words. More information can be found in [22].

The second dataset we use is the Vodafone corpus which is created by retrieving archived conversations of the Dutch Vodafone online customer service. Customers having problems with their phone, want to make contractual changes or experience other product related issues, often decide to talk with a Vodafone service agent through an online chat platform.

Every conversation that was not clearly identified as Dutch text was filtered out of the corpus using a port of Google’s Java language detection implemention 222http://code.google.com/p/language-detection/. Furthermore, to guarantee that the HRED model receives actual conversations for its training, conversations that have less than turns have also been filtered out.

The original corpus contains phone numbers, addresses, names, postal codes and other personal information. To guarantee anonymization and also to allow enough generalization, this data has been replaced by a meta-token, e.g. "<street_name>" or "<city>", which is considered to be beneficial for the performance of word embeddings. This way many more training examples will contain these general concepts (like "<street_name>") and the model can learn in which context a street name should appear. This is possible because the word embeddings of such concepts are also tuned during the training.

The final corpus was generated by using a minimal word occurrence threshold of , resulting in a dictionary size of and an average of unknowns per dialogue. The complete statistics for both datasets can be seen in Table I.

Ubuntu Vodafone
Language English Dutch
Total # of dialogues 487,337 384,897
Total # of turns 2,406,483 6,571,902
Total # of utterances 3,644,566 10,461,677
Total # of words 44,246,198 122,325,433
Avg. # of words per dialogue 90.792 317.81
Avg. # of turns per dialogue 4.938 17.07
Avg. # of words per turn 15.880 18.65
Avg. # of utterances per dialogue 7.479 27.18
Avg. # of words per utterance 11.264 11.58
TABLE I: Statistics of Ubuntu & Vodafone Corpus

Iv-B Evaluation process

A quantitative evaluation metric, the Recall@k measurement [22], has been used to compare the ranking performance of models. Given a context, a set of possible answers is presented to a model, which has to rank the answers by their likelihood of being the actual response. For a single evaluation sample, if the correct answer is ranked to be amongst the best, the model succeeded. The overall performance of a model is defined as the ratio of correctly ranked answers to all answers, i.e., the percentage of correct answers that were ranked to be amongst the best.

By iterating over the conversations in the held-out test set, an evaluation sample has been created for each individual turn or response, with the previous turns representing the context and the current turn or response being the ground truth. In addition to the actual response, a single example also contains randomly sampled answers, which the model should preferably rank lower than the true answer.

Each of the models, generative- or retrieval-based, receives the context of a conversation from an evaluation sample and has to generate or retrieve a suitable answer. The utterance-embedding of this answer is then used to compute the distance to each of the possible answers in the evaluation sample, using the cosine similarity between utterance-embeddings. The final ranking is based on this distance, placing similar answers at the top.

As we wanted to have the same conditions for all models, our ranking approach differs from the one used in [22], where answer-embeddings have been directly predicted by an additional network layer. Instead, we used Beam Search (using 5 beams) to generate an answer with the model and used the answer’s utterance embedding to compute the ranking for the generative approach.

Iv-C Results and discussion

For the Ubuntu corpus, a HRED model is trained and then the generative approach (of the original model) and the different candidate selection methods (as described in Section III-D) are compared. Results can be found in Table II.

Model R@1 R@2 R@5

Overall ranking performance of models on the ubuntu corpus. Confidence intervals (

) are shown next to the average performance.

From this Table it is obvious that AR model outperforms other candidate selection techniques as well as the generative approach. When taking into account the context of the whole conversation, results are slightly worse which means that answers are better predicted by focusing on each turn individually rather than taking into account the entire context.

For the Vodafone corpus, three HRED models have been compared, each initialized with a different set of word embeddings in order to assess the effect of local/global context in a language setting other than English. The first model, based only on local domain knowledge, received word embeddings trained with the gensim python library [23], a tool that, given a corpus, will train word embeddings specifically for that corpus. The second model utilized word embeddings from [24], representing global domain knowledge. The embeddings were trained on a corpus consisting of billion words, which was automatically generated by analyzing Dutch websites. The last model received word embeddings that are a combination of the two previously described sets. Both contain word embeddings with a feature-length of . However, the embeddings for the last model will have a length of , using a concatenation of global embeddings with features and local embeddings with features.

As with the Ubuntu corpus, the generative and retrieval based approaches (, , ) are compared and additionally in this setting, they are also tested upon different embedding initialization approaches (, and ) An overview of the results can be found in Table III.

Predicting assistant
Predicting customer
Approach Model R@1 R@2 R@5 R@1 R@2 R@5
Beam Search
Context Relevance -
Answer Relevance -
Context and Answer Relevance -
TABLE III: Overall comparison of model precisions (in ). Confidence intervals () are shown next to the average performance.

As expected, for the majority of setups, models can predict assistant responses easier than customer responses. The CAR candidate selection method outperforms all other techniques when predicting assistant responses. However, the performance of customer response prediction is slightly dominated by AR. A reason for this could be that customers often reply with new questions that might not be context related, making answer relevance more important than context relevance.

Furthermore, it can be seen that initializing the HRED model with word embeddings containing global domain knowledge results in the best performance for candidate selection approaches. However, combining global and local domain knowledge has not led to the desired improvements. This can be explained as follows: The computational graph of the HRED model that defines its training also includes tuning the word embeddings. As such, during the training, the word embeddings are already altered to encode local domain knowledge, even if they were only initialized with embeddings containing global domain knowledge. Adding additional feature-length will in the worst case only add complexity to the model.

The performance of the generative approaches, , , and , are relatively similar. However, slightly outperforms the others. This is contradictory, considering that the candidate selection methods, CR, AR and CAR, clearly perform better on embeddings generated by and (See Table III). A reason for this could be that utilizing global domain knowledge to generate an answer is more difficult than using specific domain knowledge. Especially for a very homogeneous (and domain specific) corpus, giving standard answers can work better. Nonetheless, similarity comparisons, used by the NNS approaches, could still benefit from richer embeddings.

Table IV presents some examples of answers using the Ubuntu Corpus using the generative approach (HRED) and the proposed AR model. Finally, in Table V we present one chat example from the evaluation process on Vodafone corpus (translated in English by the authors).

Context Response
I have a netbook __eou__ with 1gb of ram . Will ubuntu 12.04 be fine ?
__eou__ With unity of course !
Original: barely
AR: you don’t need much hardware to run gnome ,
unity perhaps yes though
HRED: that ’s a good idea ,
but it ’s not a good idea to stick with it
Hi Guys __eou__ I need urgent help __eou__ I’ve been trying all day to
install Ubuntu on my Mac __eou__ yeah but I can’t find ANY solution
__eou__ but can someon **unknown** me determining what to do ?
__eou__ My **unknown** USB gives me kernel panic __eou__
not syncing
Original: Why dont you use a cdrom for
installing Ubuntu ?
AR: does your Bios support booting usb device ?
HRED: not sure what to tell you , sorry
any help folks **unknown** __eou__ help with what ? __eou__
<**unknown** >hello folks , ok so i will try this again maybe
someone here is able to help me with my issue . running 12.04 on an hp
**unknown** laptop . when i plug my headphones sound still comes
from main speakers and my headphone , how can i make it so that when
i plug my headphone the sound will only play thru the headphones
Original: did you check out the **unknown**
link someone posted to you ?
AR: make sure you have the mixer and
sound devices installed in your kernel
HRED: that sounds like a hardware issue , right ?
Why can’t I install Ubuntu ? I burned the ISO to a disk , ran the disk on
windows , it **unknown** installed , but all I can get is the demo version .
And when I try to directly install the FULL version , it doesn’t do anything
but restart my computer . And the **unknown** won’t even work
unless I boot from CD .
Original: what is the ISO named ? **unknown** -
are you sure that ’s ” ubuntu ”?
AR: have you simply tried to burn a new cd ?
HRED: you have to use the alternate cd to install
the alternate cd
my computer crashed ? how can I see the log ?
Original: look in /var/log - especially at the xorg log
AR: somewhere in /var/log
HRED: it should be in the menu . lst file
Question : is sshd installed by default with ubuntu 12.04 or do i need
to install openssh
Original: normally installed by default
AR: ubuntu doesn’t install an ssh daemon by default
so yes
HRED: it should be installed by default
is there a ” terminal service ” for linux __eou__ i.e. not vnc
__eou__ X ? __eou__ ssh ? __eou__
Original: i tried it and it was rather laggy
AR: idk it just seems lame lol ; x
HRED: this is what I was looking for
How hard will it be to upgrade from Ubuntu 12.04 to 12.10 ?
Original: update-manager will prompt you if you want to
upgrade when it ’s released and you press yes
AR: it is still there , just not installed by default
HRED: first you have to use sudo apt-get dist-upgrade
to get it to work
TABLE IV: Answers of the HRED model and the retrieval-based approach (using AR) to a couple of questions from the Ubuntu corpus test set. Questions considered interesting and clean where picked by the author.
<customer> my phone is damaged,
it is new, do i get a new phone? </u>
<assistant> did the phone fall? </u>
<customer> no </u>
<assistant> when did you got the phone? </u>
<customer> I had the phone in my pocket, just when taking it out
I saw that maybe it was scratched by my keys </u> <month> </u>
Actual Response
<assistant> have you got an insurance? </u>
<assistant> I will gladly check it for you </u>
does it concern <number> ? </u>
- Candidates (best three)
<assistant> you have <number> years warranty for your device.
internal device issues are covered.
if you haven’t damaged the phone yourself,
it falls under the regular warranty. </u>
do you have an insurance with us? then you are insured either way. </u>
<assistant> here they actually can check your device
and if necessary, send it in for repair. </u>
oh … you have an insurance? </u>
<assistant> You can then send your device in for repair. </u>
if you have the device replacement service in your subscription
your device will be <unk> and you will receive a replacement device.
In case you don’t have it? </u>
then you can bring your device </u> to a Vodafone shop. </u>
TABLE V: An example showing the three best responses retrieved by the - model to a given context. For comparison reasons, the answer generated by the model and the actual response are shown as well.

V Conclusion and Future Work

End-to-End Dialogue Systems are relatively new and most architectures are far away from being ready for deployment in actual industry that most likely will require more years of research. The architecture proposed in this paper can be seen as a combination of end-to-end and modular Dialogue System. The used retrieval based approach, utilizing dialogue and utterance embeddings which were trained end-to-end, has been shown to outperform the generative approach of the HRED model.

More recently proposed end-to-end systems, the VHRED model and Multiresolution Recurrent Neural Networks [25], both being an improved version of the HRED model, are raising another question: Will one of these models outperform the proposed retrieval based approach, even though all operate on the same embeddings, i.e., at which point does the generative approach benefit from the embeddings’ quality more than the retrieval-based? Dialogue and utterance embeddings have been only generated by one source, the HRED model. For comparison reasons, it would be interesting to explore the performance of other encoding approaches, such as averaging over word embeddings. Additionally, the embeddings generated by the recently proposed VHRED model and Multiresolution Recurrent Neural Networks are expected to be of higher quality and likely to improve the performance of the proposed approach. However, one must not underestimate the importance of the proposed ranking system, since it can directly be used by a human agent as a means to assist communication with a client. An interesting research direction arising from this paper would be to allow human agents to affect the ranking score and by this way providing feedback (in terms of reinforcement learning [26]) to the system, which then might be able to re-rank answers.

Another future direction is the simulation of conversations with a tree search, by representing context embeddings as states (tree nodes) and utterance embeddings as actions (connections between nodes). This technique is inspired by game simulations where the desirable state is found through an exploration/exploitation strategy. Using responses retrieved by NNS as a set of actions, the search tree can explore possible paths and score responses based on the quality of simulated conversations.

Moreover, the candidate selection module of our proposed pipeline reveals new opportunities for utilizing such a ranking/similarity model in other problems/domains such as recommender systems. Reviews of products, services, etc. could be encoded using a model like HRED and then based on a query (question) of a user, recommendation can take place by ranking the most relevant reviews (answers).

Finally, another research direction towards the implementation of such a dialogue system, is the utilization of additional context information a service agent can see and that is not contained in the conversation. Considering the size of corpora used to train end-to-end systems (usually around conversations), manual annotation can be very slow and costly. Finding an automated approach to make addresses, contractual details and other features accessible to an end-to-end Dialogue System is an interesting and rewarding task. Solving (some of) the aforementioned problems will facilitate the deployment of end-to-end Dialogue Systems in online chat service environments, improving robustness, utility and customer experience.


We would like to thank Harry Beckers and Marcel Overdijk for their collaboration and support. We gratefully acknowledge the support of QNH Consulting with the donation of the Nvidia GTX 1070 GPU used for this research and for providing the Vodafone dataset.


  • [1] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Advances in neural information processing systems, 2013, pp. 3111–3119.
  • [2] I. V. Serban, A. Sordoni, Y. Bengio, A. Courville, and J. Pineau, “Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models,” in Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence.   AAAI Press, 2016, pp. 3776–3783.
  • [3] I. V. Serban, A. Sordoni, R. Lowe, L. Charlin, J. Pineau, A. Courville, and Y. Bengio, “A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues,” arXiv preprint arXiv:1605.06069v3, 2016.
  • [4] A. Shawar and E. Atwell, “Chatbots: are they really useful?” in Journal for Language Technology and Computational Linguistics, vol. 22, no. 1.   GSCL German Society for Computational Linguistics, 2007, pp. 29–49.
  • [5] D. Jurafsky and H. James,

    Speech and language processing an introduction to natural language processing, computational linguistics, and speech

    .   Pearson Education, 2000.
  • [6] M. Bates, “Models of natural language understanding,” in Proceedings of the National Academy of Sciences of the United States of America, vol. 92.   National Academy Press, 1995, pp. 9977–9982.
  • [7] C. D. Manning and H. Schütze, “Foundations of statistical natural language processing,” The MIT Press, 1999.
  • [8] J. Dowding, J. M. Gawron, D. Appelt, J. Bear, L. Cherny, R. Moore, and D. Moran, “Gemini: A natural language system for spoken-language understanding,” in Proceedings of the 31st annual meeting on Association for Computational Linguistics.   Association for Computational Linguistics, 1993, pp. 54–61.
  • [9] R. Collobert and J. Weston, “A unified architecture for natural language processing: Deep neural networks with multitask learning,” in Proceedings of the 25th international conference on Machine learning.   ACM, 2008, pp. 160–167.
  • [10] Y. LeCun, Y. Bengio, and G. E. Hinton, “Deep learning,” in Nature: International weekly journal of science, vol. 521, no. 7553.   Macmillan, 2015, pp. 436–444.
  • [11] O. Vinyals and Q. Le, “A neural conversational model,” in International Conference on Machine Learning: Deep Learning Workshop, 2015.
  • [12] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Advances in neural information processing systems, 2014, pp. 3104–3112.
  • [13] P. Shah, D. Hakkani-Tür, and L. Heck, “Interactive reinforcement learning for task-oriented dialogue management,” in NIPS 2016 Deep Learning for Action and Interaction Workshop, 2016.
  • [14] S. Young, M. Gašić, B. Thomson, and J. D. Williams, “POMDP-based Statistical Spoken Dialogue Systems: a Review,” in Proceedings of the IEEE, vol. 101, no. 5, 2013, pp. 1160–1179.
  • [15] J. Schmidhuber, “Deep learning in neural networks: An overview,” in Neural networks, vol. 61.   Elsevier, 2015, pp. 85–117.
  • [16] Y. Bengio, N. Boulanger-Lewandowski, and R. Pascanu, “Advances in optimizing recurrent networks,” in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on.   IEEE, 2013, pp. 8624–8628.
  • [17] T.-H. Wen, M. Gašic, D. Kim, N. Mrkšic, P.-H. Su, D. Vandyke, and S. Young, “Stochastic Language Generation in Dialogue using Recurrent Neural Networks with Convolutional Sentence Reranking,” in 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2015, p. 275.
  • [18] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” in Proceedings of the Empiricial Methods in Natural Language Processing.   Association for Computational Linguistics, 2014, pp. 1724–1734.
  • [19] R. Weber, H.-J. Schek, and S. Blott, “A quantitative analysis and performance study for similarity-search methods in high-dimensional spaces,” in Very Large Data Bases (VLDB) Conference, vol. 98, 1998, pp. 194–205.
  • [20] S. Har-Peled, P. Indyk, and R. Motwani, “Approximate Nearest Neighbor: Towards Removing the Curse of Dimensionality.” in Theory of computing, vol. 8, no. 1, 2012, pp. 321–350.
  • [21] M. Bawa, T. Condie, and P. Ganesan, “LSH Forest: self-tuning indexes for similarity search,” in Proceedings of the 14th international conference on World Wide Web.   ACM, 2005, pp. 651–660.
  • [22] R. Lowe, N. Pow, I. V. Serban, and J. Pineau, “The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems,” in 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2015.
  • [23] R. Řehůřek and P. Sojka, “Software Framework for Topic Modelling with Large Corpora,” in Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks.   Valletta, Malta: ELRA, May 2010, pp. 45–50, http://is.muni.cz/publication/884893/en.
  • [24] S. Tulkens, C. Emmery, and W. Daelemans, “Evaluating Unsupervised Dutch Word Embeddings as a Linguistic Resource,” in Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), N. C. C. Chair), K. Choukri, T. Declerck, M. Grobelnik, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, and S. Piperidis, Eds.   Paris, France: European Language Resources Association (ELRA), may 2016.
  • [25] I. V. Serban, T. Klinger, G. Tesauro, K. Talamadupula, B. Zhou, Y. Bengio, and A. Courville, “Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation,” arXiv preprint arXiv:1606.00776, 2016.
  • [26] J. D. Williams and G. Zweig, “End-to-end lstm-based dialog control optimized with supervised and reinforcement learning,” arXiv preprint arXiv:1606.01269, 2016.