Text-only based Dialogue systems, also called Conversational Agents, Chatbots or Chatterbots, have become very popular in the research community and for large companies. The reason for the rise in popularity lies in the fact that their ability to interact intelligently with humans has improved significantly due to advancements in hardware technologies and Artificial Intelligence.
One of the latest effective approaches 
is to represent words, phrases, or even complete dialogues as fixed-length vectors of floating point numbers, also called embeddings (or distributed representations or feature vectors). The Hierarchical Recurrent Encoder-Decoder (HRED) and its successors , (as well as similar related models) are specifically designed to encode the meaning of textual conversations regarding the special structure that originates from multiple turn-taking speakers.
In our approach, a context embedding, a vector encoding the meaning of a conversation up to a certain time step , encoded by the HRED model, serves as input to the decoder component to generate a textual answer. We explore the performance of a retrieval-based model that uses the utterance- and context-embeddings, previously generated by the HRED model, to find similar conversations and rank possible candidate answers. We argue that a retrieval-based approach, based on embeddings, can outperform the generative approach, as the retrieval of similar conversations is less dependent on high quality embeddings and less susceptible to poorly trained embeddings.
The rest of the paper is organized as follows: We first give an outline of the research around dialogue systems, showing the development from purely script-based systems to deep networks generating answers end-to-end. The proposed pipeline is discussed in Section III
. Experimental setup (datasets, evaluation metrics, models implemented and compared) as well as the results are discussed in SectionIV. Finally, we conclude the paper by summarizing the main findings and outline future work.
Ii Related Work
The purpose of Dialogue Systems (DS), often also termed Conversational Agents (CA) or Chatterbots, is to converse with humans to provide information, help in decision making, perform administrative services, or just for the sake of entertainment . The traditional design of Dialogue Systems  follows a modular approach, splitting the system usually into a Natural Language Understanding (NLU) module, a Dialogue Manager and a Natural Language Generation (NLG) unit. The NLU module processes the raw user input and extracts useful information and features that can be used by the Dialogue Manager to update internal states, send queries to a knowledge base or, more generally, find actions based on a script. The NLG acts inversely to the NLU module, receiving features and information from the Dialogue Manager to generate a response that, finally, will be presented to the user.
One of the simplest design approaches for an NLU is to simply spot certain key-words or combinations of them. This is often the general procedure of script-based chatbots and the approach followed by ELIZA. However, there is a long history of attempts 
to improve NLU and find better representations of text. With advances in machine learning, the development ranges from statistical modelling of language, semantic parsing , skip-gram models , and others, to approaches utilizing deep neural architectures [9, 10]
. Neural networks have also been used to improve NLG[11, 12]
. Recently, Dialogue Managers have made similar advances towards automated solutions, with a focus on reinforcement learning[13, 14], generating policies of how to interact with humans, based on some state representation.
With the rise of Deep Learning (DL) in recent years 
and an increasing company interest in chatterbots, end-to-end Dialogue Systems, such as deep Recurrent Neural Networks, constituting all modules in one model, have become one of the major research topics for Dialogue Systems. The tasks of NLU, NLG, and the Dialogue Manager are performed by a single deep network that is trained to reproduce conversations from a large dataset. Such a system would generate an answer end-to-end from raw user input. Even though training deep RNNs can be considerably difficult , only one model would need to be optimized, and one could benefit from the neural model’s capability to generate natural responses [17, 11].
Our proposed pipeline is a combination of a generative- and retrieval-based approach. An encoder model, such as the HRED model (or could be one of its more advanced variations) is trained end-to-end on a textual corpus, using an objective function that is based on how capable the model is of generating the answers in the training set. After the training however, the decoder component of the HRED model is not used to generate answers. Instead, we argue that a retrieval-based approach taking over the NLG part performs equally or better in both general and specific domains.
Iii Model description
The proposed model can be split up into three individual components. The first component, the encoder, utilizes the HRED model to encode raw conversations into embeddings containing the actual meaning. The second component, a retrieval-based approach using an Approximate Nearest Neighbor (ANN) model, is responsible for retrieving similar conversations from a database of embedding- and raw-text-tuples. Given the context of an unfinished conversation, suitable responses are considered to be contained in similar conversations, retrieved by the ANN model. The last model component receives a retrieved set and ranks possible answers based on answer- and context-relevance. The entire pipeline can be seen in Figure 1.
Iii-a Gated Recurrent Unit (GRU)
Recurrent Neural Networks (RNNs) have been designed to process sequential data by encoding historic information into a hidden state. When optimizing the network to predict future values using the current input and historic values, the hidden state naturally becomes a new representation form of the processed data. LSTMs, first introduced 1997, are an attempt to increase a RNN’s capabilities to remember long-term dependencies by replacing hidden units with more complex memory cells, capable of controlling the information flow in and out of the cells.
The more recently proposed GRUs are similar to LSTM cells aiming to improve a RNN’s capabilities to remember long-term dependencies. However, they use a different gate design, have fewer parameters to train and come without an additional cell state. Two gates, the reset and update gates and , operate directly on the hidden state, i.e., the hidden layer. Parametrized by , and , while conditioned on the current input and previous result , GRU gate vectors are computed as:
being the sigmoid function. The update gatecombines the function of the input and forget gate by controlling how much the new hidden state (
) is defined by either the current input or the last hidden state, using linear interpolation:
with being the candidate activation. The reset gate is used to calculate , controlling similarly how much of the previous hidden state to keep:
with being the hyperbolic tangent function.
Iii-B Hierarchical Recurrent Encoder-Decoder
The HRED model essentially consists of three stacked RNNs: the utterance encoder, context encoder, and utterance decoder, each of them depending on the result of its predecessor and operating on distributed representations.
Formally, a dialogue , the input to such a model, can be represented as a sequence of utterances , with being a sequence of word indices , each of them usually pointing to a vocabulary reference or directly to a word embedding. These become the input to the utterance encoder.
Iii-B1 Encoding steps
computing current hidden state , conditioned on the previous, and on current input . To comply with the HRED notation, the utterance embedding of the current utterance , including word , is calculated as:
Applying equation 5 consecutively on word embeddings , results in an equally-sized set of hidden states , where the last hidden state is the summary of all words in the same utterance. As such, we denote to be the hidden state that represents utterance .
Using this encoding approach, a set of utterances is encoded into hidden states . Those are used as input to the GRU-based context encoder, similar to how word embeddings acted as input to the utterance encoder. As such, context embeddings are a summary of utterances and represent entire dialogues. They are computed as:
Iii-B2 Decoding step
In addition to encoding a sequence of embeddings into a hidden state, the decoder component generates word probabilities over a vocabulary, given some contextand previous words .
Firstly, to condition the decoder RNN on previous utterances, the initialization of its hidden state is based on the context encoders last hidden state . If not designed explicitly, context and decoder RNN usually have different hidden state dimensionalities, which is why an additional network layer is added to project context embeddings into the decoder space:
with parameters and and being the decoder RNN’s initial hidden state.
Given a set of words , having been previously generated or representing a training example, the decoder RNN hidden state is similarly computed as it was done for the encoder RNNs:
processing words consecutively. The first iteration uses the hidden state computed by equation 7 and a zero-value embedding for to predict the first word of an utterance.
Using both, the hidden state and word embedding of , the word embedding of current word is then predicted as:
with the additional parameters , and . and control which part of the previous context- and word embedding contribute to the new word embedding and how much of that part is used.
By calculating the dot product of such generated word embeddings with the embeddings in a vocabulary, one can compute the similarity between prediction and existing words, with the most similar word being the most likely one. The actual probability of a word occurring next is based on this similarity:
with being the word embedding of word , the predicted word embedding and the vocabulary size.
By computing probabilities for each word in the vocabulary, one can create a distribution from which words can be sampled. Pushing sampled words back into the decoder allows to generate the next word, extending an utterance until an end-of-sequence meta token has been reached. Possible generations can be explored using probability based search techniques such as Beam Search .
Iii-C Retrieval Model
Using the encoded corpus as a database of vectors, a Nearest Neighbor Search (NNS) algorithm can be used to find close embeddings in the whole set. For this purpose, an ANN approach has been considered, as general space-partitioning approaches, aiming to improve the NNS performance, suffer from the curse of dimensionality.
Locality Sensitive Hashing (LSH) [20, 21] is an ANN approach that uses a set of hashing functions to project similar data points into buckets and as such, significantly restricts the search space to the size of the bucket. For a projection, a binary string label is constructed by applying different hashing functions to a single data point, where the output of such a function is either one or zero. The desired goal of a hashing function is to output the same label for similar data points and differing labels for dissimilar ones. Therefore, binary string labels that are similar, indicate that also the original data points are similar. The string label is then used as a key to index a bucket of similar data points, where a brute force approach can be applied on a much smaller set. A collection of buckets is called a hash-table and tables constitute the entire model.
One of the main issues with the basic LSH algorithm  is that choosing the optimal number of hashing functions and number of tables requires one to know the most suitable value for , the threshold separating similar and dissimilar points. The LSH-Forest algorithm solves this issue by allowing labels with variable length and thus, eliminating parameter .
Instead of linking fixed-length labels to buckets, the label string is stored in a prefix tree, a binary tree (also called ’trie’), in which keys are not contained within nodes but derived from the path that leads from the root to a node.
For the case that two points are very similar or equal, the length-limiting parameter prevents their labels from growing too large.
Each level of the tree is associated with a different hashing function, sampled uniformly and with replacement from a family of hashing functions . Such a tree, an LSH-Tree, is the equivalent to an LSH-based hash-table and the composition of trees is an LSH-Forest.
Given a query point , finding close neighbors in a set of LSH-Trees is performed in two phases. First, in a top-down phase or descent, each tree is searched for the leaf node with the best match to the binary string label of . Labels are computed individually for each tree, starting from the root and extending the label until a leaf node is reached.
Inspecting the matches from all trees, the match with the longest prefix defines the tree-level from which the bottom-up accumulation, the second phase, begins.
Iii-D Candidate Selection
Given a query context of an unfinished conversation, using the previously discussed LSH-Forest algorithm, one can retrieve a candidate set from a database of encoded conversations. Candidate answers will be scored based on the matching degree between the retrieved and the original context in terms of question-to-question similarity or answer relevance or other text-based features. However, the scoring functions introduced in this section will solely be based on vector comparison metrics, such as the cosine similarity, as text-based comparison is less rewarding and more difficult and tedious to implement. For the sake of clarity, the query context embedding is defined as, the textual candidates as , the context embeddings of candidates as , and the utterance embeddings of candidates as .
Iii-D1 Context Relevance
The similarity of two conversations or the distance between a query context and a candidate context has, intuitively, a big impact on the retrieved answer, i.e, the more two questions are similar, the higher the probability that the answers are similar as well. If the cosine similarity has been chosen as the distance function , the labels returned by the nearest neighbor search are already sorted by context-to-context distance. Formally, given a candidate response and a query context , the Context Relevance (CR) cost function is defined as:
Iii-D2 Answer Relevance
By manual inspection of near neighbors, it became apparent that the correct answer is usually represented or almost captured in many topic-related candidates. Assuming that the most suitable topic for answering is dominantly represented amongst candidates, responses are ranked based on how much they capture the general topic. Formally, the cost of a response is defined by the accumulated similarity between its respective embedding and the utterance embeddings of all other candidates (See Figure 3), normalized by length :
Iii-D3 Combining Context and Answer Relevance
The problem with the previous approach is that the candidates that are off-topic still contribute to the answer relevance cost. Therefore, in a pre-step, according to the previously described context relevance metric, the top candidates are accumulated to represent the best general answer topic. In the next step, candidates are ranked based on their similarity to these responses. Formally, combined Context and Answer Relevance (CAR) is defined as:
The first dataset we use is the Ubuntu Dialogue Corpus which has been studied in most state of the art systems (similar to HRED). The ubuntu dataset contains almost 1 million multi-turn dialogues, with a total of over 7 million utterances and 100 million words. More information can be found in .
The second dataset we use is the Vodafone corpus which is created by retrieving archived conversations of the Dutch Vodafone online customer service. Customers having problems with their phone, want to make contractual changes or experience other product related issues, often decide to talk with a Vodafone service agent through an online chat platform.
Every conversation that was not clearly identified as Dutch text was filtered out of the corpus using a port of Google’s Java language detection implemention 222http://code.google.com/p/language-detection/. Furthermore, to guarantee that the HRED model receives actual conversations for its training, conversations that have less than turns have also been filtered out.
The original corpus contains phone numbers, addresses, names, postal codes and other personal information. To guarantee anonymization and also to allow enough generalization, this data has been replaced by a meta-token, e.g. "<street_name>" or "<city>", which is considered to be beneficial for the performance of word embeddings. This way many more training examples will contain these general concepts (like "<street_name>") and the model can learn in which context a street name should appear. This is possible because the word embeddings of such concepts are also tuned during the training.
The final corpus was generated by using a minimal word occurrence threshold of , resulting in a dictionary size of and an average of unknowns per dialogue. The complete statistics for both datasets can be seen in Table I.
|Total # of dialogues||487,337||384,897|
|Total # of turns||2,406,483||6,571,902|
|Total # of utterances||3,644,566||10,461,677|
|Total # of words||44,246,198||122,325,433|
|Avg. # of words per dialogue||90.792||317.81|
|Avg. # of turns per dialogue||4.938||17.07|
|Avg. # of words per turn||15.880||18.65|
|Avg. # of utterances per dialogue||7.479||27.18|
|Avg. # of words per utterance||11.264||11.58|
Iv-B Evaluation process
A quantitative evaluation metric, the Recall@k measurement , has been used to compare the ranking performance of models. Given a context, a set of possible answers is presented to a model, which has to rank the answers by their likelihood of being the actual response. For a single evaluation sample, if the correct answer is ranked to be amongst the best, the model succeeded. The overall performance of a model is defined as the ratio of correctly ranked answers to all answers, i.e., the percentage of correct answers that were ranked to be amongst the best.
By iterating over the conversations in the held-out test set, an evaluation sample has been created for each individual turn or response, with the previous turns representing the context and the current turn or response being the ground truth. In addition to the actual response, a single example also contains randomly sampled answers, which the model should preferably rank lower than the true answer.
Each of the models, generative- or retrieval-based, receives the context of a conversation from an evaluation sample and has to generate or retrieve a suitable answer. The utterance-embedding of this answer is then used to compute the distance to each of the possible answers in the evaluation sample, using the cosine similarity between utterance-embeddings. The final ranking is based on this distance, placing similar answers at the top.
As we wanted to have the same conditions for all models, our ranking approach differs from the one used in , where answer-embeddings have been directly predicted by an additional network layer. Instead, we used Beam Search (using 5 beams) to generate an answer with the model and used the answer’s utterance embedding to compute the ranking for the generative approach.
Iv-C Results and discussion
For the Ubuntu corpus, a HRED model is trained and then the generative approach (of the original model) and the different candidate selection methods (as described in Section III-D) are compared. Results can be found in Table II.
Overall ranking performance of models on the ubuntu corpus. Confidence intervals () are shown next to the average performance.
From this Table it is obvious that AR model outperforms other candidate selection techniques as well as the generative approach. When taking into account the context of the whole conversation, results are slightly worse which means that answers are better predicted by focusing on each turn individually rather than taking into account the entire context.
For the Vodafone corpus, three HRED models have been compared, each initialized with a different set of word embeddings in order to assess the effect of local/global context in a language setting other than English. The first model, based only on local domain knowledge, received word embeddings trained with the gensim python library , a tool that, given a corpus, will train word embeddings specifically for that corpus. The second model utilized word embeddings from , representing global domain knowledge. The embeddings were trained on a corpus consisting of billion words, which was automatically generated by analyzing Dutch websites. The last model received word embeddings that are a combination of the two previously described sets. Both contain word embeddings with a feature-length of . However, the embeddings for the last model will have a length of , using a concatenation of global embeddings with features and local embeddings with features.
As with the Ubuntu corpus, the generative and retrieval based approaches (, , ) are compared and additionally in this setting, they are also tested upon different embedding initialization approaches (, and ) An overview of the results can be found in Table III.
|Context and Answer Relevance||-|
As expected, for the majority of setups, models can predict assistant responses easier than customer responses. The CAR candidate selection method outperforms all other techniques when predicting assistant responses. However, the performance of customer response prediction is slightly dominated by AR. A reason for this could be that customers often reply with new questions that might not be context related, making answer relevance more important than context relevance.
Furthermore, it can be seen that initializing the HRED model with word embeddings containing global domain knowledge results in the best performance for candidate selection approaches. However, combining global and local domain knowledge has not led to the desired improvements. This can be explained as follows: The computational graph of the HRED model that defines its training also includes tuning the word embeddings. As such, during the training, the word embeddings are already altered to encode local domain knowledge, even if they were only initialized with embeddings containing global domain knowledge. Adding additional feature-length will in the worst case only add complexity to the model.
The performance of the generative approaches, , , and , are relatively similar. However, slightly outperforms the others. This is contradictory, considering that the candidate selection methods, CR, AR and CAR, clearly perform better on embeddings generated by and (See Table III). A reason for this could be that utilizing global domain knowledge to generate an answer is more difficult than using specific domain knowledge. Especially for a very homogeneous (and domain specific) corpus, giving standard answers can work better. Nonetheless, similarity comparisons, used by the NNS approaches, could still benefit from richer embeddings.
Table IV presents some examples of answers using the Ubuntu Corpus using the generative approach (HRED) and the proposed AR model. Finally, in Table V we present one chat example from the evaluation process on Vodafone corpus (translated in English by the authors).
|my computer crashed ? how can I see the log ?||
|How hard will it be to upgrade from Ubuntu 12.04 to 12.10 ?||
|<assistant> have you got an insurance? </u>|
|<assistant> I will gladly check it for you </u>|
|does it concern <number> ? </u>|
|- Candidates (best three)|
V Conclusion and Future Work
End-to-End Dialogue Systems are relatively new and most architectures are far away from being ready for deployment in actual industry that most likely will require more years of research. The architecture proposed in this paper can be seen as a combination of end-to-end and modular Dialogue System. The used retrieval based approach, utilizing dialogue and utterance embeddings which were trained end-to-end, has been shown to outperform the generative approach of the HRED model.
More recently proposed end-to-end systems, the VHRED model and Multiresolution Recurrent Neural Networks , both being an improved version of the HRED model, are raising another question: Will one of these models outperform the proposed retrieval based approach, even though all operate on the same embeddings, i.e., at which point does the generative approach benefit from the embeddings’ quality more than the retrieval-based? Dialogue and utterance embeddings have been only generated by one source, the HRED model. For comparison reasons, it would be interesting to explore the performance of other encoding approaches, such as averaging over word embeddings. Additionally, the embeddings generated by the recently proposed VHRED model and Multiresolution Recurrent Neural Networks are expected to be of higher quality and likely to improve the performance of the proposed approach. However, one must not underestimate the importance of the proposed ranking system, since it can directly be used by a human agent as a means to assist communication with a client. An interesting research direction arising from this paper would be to allow human agents to affect the ranking score and by this way providing feedback (in terms of reinforcement learning ) to the system, which then might be able to re-rank answers.
Another future direction is the simulation of conversations with a tree search, by representing context embeddings as states (tree nodes) and utterance embeddings as actions (connections between nodes). This technique is inspired by game simulations where the desirable state is found through an exploration/exploitation strategy. Using responses retrieved by NNS as a set of actions, the search tree can explore possible paths and score responses based on the quality of simulated conversations.
Moreover, the candidate selection module of our proposed pipeline reveals new opportunities for utilizing such a ranking/similarity model in other problems/domains such as recommender systems. Reviews of products, services, etc. could be encoded using a model like HRED and then based on a query (question) of a user, recommendation can take place by ranking the most relevant reviews (answers).
Finally, another research direction towards the implementation of such a dialogue system, is the utilization of additional context information a service agent can see and that is not contained in the conversation. Considering the size of corpora used to train end-to-end systems (usually around conversations), manual annotation can be very slow and costly. Finding an automated approach to make addresses, contractual details and other features accessible to an end-to-end Dialogue System is an interesting and rewarding task. Solving (some of) the aforementioned problems will facilitate the deployment of end-to-end Dialogue Systems in online chat service environments, improving robustness, utility and customer experience.
We would like to thank Harry Beckers and Marcel Overdijk for their collaboration and support. We gratefully acknowledge the support of QNH Consulting with the donation of the Nvidia GTX 1070 GPU used for this research and for providing the Vodafone dataset.
-  T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Advances in neural information processing systems, 2013, pp. 3111–3119.
-  I. V. Serban, A. Sordoni, Y. Bengio, A. Courville, and J. Pineau, “Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models,” in Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. AAAI Press, 2016, pp. 3776–3783.
-  I. V. Serban, A. Sordoni, R. Lowe, L. Charlin, J. Pineau, A. Courville, and Y. Bengio, “A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues,” arXiv preprint arXiv:1605.06069v3, 2016.
-  A. Shawar and E. Atwell, “Chatbots: are they really useful?” in Journal for Language Technology and Computational Linguistics, vol. 22, no. 1. GSCL German Society for Computational Linguistics, 2007, pp. 29–49.
D. Jurafsky and H. James,
Speech and language processing an introduction to natural language processing, computational linguistics, and speech. Pearson Education, 2000.
-  M. Bates, “Models of natural language understanding,” in Proceedings of the National Academy of Sciences of the United States of America, vol. 92. National Academy Press, 1995, pp. 9977–9982.
-  C. D. Manning and H. Schütze, “Foundations of statistical natural language processing,” The MIT Press, 1999.
-  J. Dowding, J. M. Gawron, D. Appelt, J. Bear, L. Cherny, R. Moore, and D. Moran, “Gemini: A natural language system for spoken-language understanding,” in Proceedings of the 31st annual meeting on Association for Computational Linguistics. Association for Computational Linguistics, 1993, pp. 54–61.
-  R. Collobert and J. Weston, “A unified architecture for natural language processing: Deep neural networks with multitask learning,” in Proceedings of the 25th international conference on Machine learning. ACM, 2008, pp. 160–167.
-  Y. LeCun, Y. Bengio, and G. E. Hinton, “Deep learning,” in Nature: International weekly journal of science, vol. 521, no. 7553. Macmillan, 2015, pp. 436–444.
-  O. Vinyals and Q. Le, “A neural conversational model,” in International Conference on Machine Learning: Deep Learning Workshop, 2015.
-  I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Advances in neural information processing systems, 2014, pp. 3104–3112.
-  P. Shah, D. Hakkani-Tür, and L. Heck, “Interactive reinforcement learning for task-oriented dialogue management,” in NIPS 2016 Deep Learning for Action and Interaction Workshop, 2016.
-  S. Young, M. Gašić, B. Thomson, and J. D. Williams, “POMDP-based Statistical Spoken Dialogue Systems: a Review,” in Proceedings of the IEEE, vol. 101, no. 5, 2013, pp. 1160–1179.
-  J. Schmidhuber, “Deep learning in neural networks: An overview,” in Neural networks, vol. 61. Elsevier, 2015, pp. 85–117.
-  Y. Bengio, N. Boulanger-Lewandowski, and R. Pascanu, “Advances in optimizing recurrent networks,” in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 8624–8628.
-  T.-H. Wen, M. Gašic, D. Kim, N. Mrkšic, P.-H. Su, D. Vandyke, and S. Young, “Stochastic Language Generation in Dialogue using Recurrent Neural Networks with Convolutional Sentence Reranking,” in 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2015, p. 275.
-  K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” in Proceedings of the Empiricial Methods in Natural Language Processing. Association for Computational Linguistics, 2014, pp. 1724–1734.
-  R. Weber, H.-J. Schek, and S. Blott, “A quantitative analysis and performance study for similarity-search methods in high-dimensional spaces,” in Very Large Data Bases (VLDB) Conference, vol. 98, 1998, pp. 194–205.
-  S. Har-Peled, P. Indyk, and R. Motwani, “Approximate Nearest Neighbor: Towards Removing the Curse of Dimensionality.” in Theory of computing, vol. 8, no. 1, 2012, pp. 321–350.
-  M. Bawa, T. Condie, and P. Ganesan, “LSH Forest: self-tuning indexes for similarity search,” in Proceedings of the 14th international conference on World Wide Web. ACM, 2005, pp. 651–660.
-  R. Lowe, N. Pow, I. V. Serban, and J. Pineau, “The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems,” in 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2015.
-  R. Řehůřek and P. Sojka, “Software Framework for Topic Modelling with Large Corpora,” in Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. Valletta, Malta: ELRA, May 2010, pp. 45–50, http://is.muni.cz/publication/884893/en.
-  S. Tulkens, C. Emmery, and W. Daelemans, “Evaluating Unsupervised Dutch Word Embeddings as a Linguistic Resource,” in Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), N. C. C. Chair), K. Choukri, T. Declerck, M. Grobelnik, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, and S. Piperidis, Eds. Paris, France: European Language Resources Association (ELRA), may 2016.
-  I. V. Serban, T. Klinger, G. Tesauro, K. Talamadupula, B. Zhou, Y. Bengio, and A. Courville, “Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation,” arXiv preprint arXiv:1606.00776, 2016.
-  J. D. Williams and G. Zweig, “End-to-end lstm-based dialog control optimized with supervised and reinforcement learning,” arXiv preprint arXiv:1606.01269, 2016.