Chatbots have become more and more popular over the past years, owing for instance to the success of text messaging apps, advances in natural language processing techniques, and advances in scalability. Solutions based on chatbots allow industry services to be 24-hours a day available to their customers, cutting expenses, and automating many of their business processes(forbes, ). Chatbots can be task-oriented, designed to a particular task to get information from the user to help complete the task (e.g., booking airline flights) or can be systems designed for extended conversations, mimicking of human-human interaction (e.g., psychological counseling) (chatbot_book, ). A by-product of that is the increasing volume of data, in special textual dialogs, that is being generated by systems making use of chatbots. There is demand for approaches to better understanding such data to either gather insights from the users or to improve the chatbots since data analytics for chatbots is still an area that needs more investigation.
In such sense, one kind of analysis that could help understanding chatbot data is to find similarity between dialogs. Therefore, grouping similar dialogs can give not only insights about the customers that interact with the chatbot but also can help on the creation corpus-based chatbots that are built mining conversations between human-human or human-machine conversations (serban-2015-survey, )
. Finding similar objects is a traditional task in several areas as content-based image retrieval(liu2007survey, ), text similarity identification (hotho2005brief, ), code plagiarism detection (maurer2006plagiarism, ), song identification (typke2005survey, ) and so on. For dialogs, such similarity-based retrieval could be useful to recover dialogs that present some similarity either in content or structure, and the identification of similar sets of dialogs can be valuable to better understand the data and further improve the content or structure of the chatbots.
Most of the work in dialog aims at studying specific details of the dialog or the users with the goal of, to cite a few, identify genders (kose2007mining, )
, classify speech acts(carpenter2011role, ; ferschke2012behind, ; schabus2016data, ), identify socio-cultural phenomena (strzalkowski2010modeling, ), understand user interaction (khan2002mining, ), study linguistic coordination (danescu2011chameleons, ), or predict emotions (litman2004predicting, ), (maeireizo2004co, ). Another front can be observed in the efforts to build corpora for dialog systems (serban2015survey, ), dialog generation (walker2012annotated, ; serban2017multiresolution, ; serban2017hierarchical, ), dialog generation evaluation (liu2016not, ), dialog control (williams2016end, ) and build specific corpus such as Ubuntu (lowe2015ubuntu, ) or modeling Twitter as a dialog conversation (ritter2010unsupervised, ). However, none of the supra-cited works address dialog similarity. A lot of works in dialog is focus on spoken dialog, to fill gaps (mesnil2017, ), control dialog (williams2017hybrid, ; bordes2016learning, ; wen2016network, ) and measure sucess (su2015learning, ).
One way to compare two dialogs is to solely consider the textual content of the dialog and then use some standard text similarity technique. Text similarity is a well-known task when people mine large volume of text data, and several techniques have been developed (metzler2007similarity, ; gomaa2013survey, ). Nevertheless, data from chatbots or dialogs present also a structure that goes beyond traditional text data. Chatbots data present some specific characteristics such as the order and the time interval of the utterances, the user interactions, speech acts, turns and so on. For that reason, computing dialog similarity becomes a complex task than traditional text similarity.
In this paper, we propose a similarity metric based on both textual content and structural properties of dialogs. To evaluate our proposed metric, we perform experiments on the well-know Switchboard dataset (SWITCHBOARD92, ). To the best of our knowledge, we are the first to address the problem of dialog similarity and thus, in this paper, we will present:
A new metric for detecting similarity between dialogs;
New metrics that combine metrics from dialog structure and metrics from the text to better represent dialogs;
Experiments showing that these new metrics works better for dialogs them only text;
A discussion on the challenges of this new area and the next steps to be taken.
2. Dialog Similarity
Two dialogs can be similar considering different aspects. For instance, they can be similar in terms of textual content, that is, they address the same topics and/or the same entities, and thus forth. On the other hand, dialogs can also be similar in terms of structural features, such as the length of the dialog or the presence/absence of some kind of identified pattern of interaction.
Our method combines evidences from both textual content and structural features to compute similarity between two dialogs. Considering these evidences as complementary, we combine them based on computing the similarity of content and structure individually and combine their similarity results with the Borda Count method at ranking level, so that we avoid the burden of dealing with different scales of the features or distances if combined differently. .
2.1. Textual-based Similarity
For computing the features related to similarity based on textual content, we simply convert the dialog to a free-form text, where each dialog utterance consists of a new sentence. Then, we apply standard text mining techniques, such as stop word removal methods and the computation of Term Frequency Inverse Document Frequency (TF-IDF) measure (Weiss2012, ), being defined as:
where is the th word in the dictionary, is its frequency in the documents, is the number of documents in the dataset, and is the number of documents in which the word appears.
Once those features are computed, the cosine distance is used to compute the similarity between the TF-IDF vector of two dialogs, as:
2.2. Dialog Structural Similarity
For the similarity based on structural features of the dialog, we considered information about the turns, in which each turn is the contribution of one a dialog participant. We also compute metrics related to conversation cycles. A cycle is a situation when one participant of the conversation has to rephrase his/her idea until it is clear for the other participant or he/she gives up on that topic. For example, if you have the following conversation to book a plane ticket (Table 1).
The conversation above has the repetition of the same intent which is related to book a ticket. The participant has to rephrase three times the question, therefore there is a cycle of size four in the dialog. Our approach estimates the size of cycle computing the similarity of two turns, particularly we use cosine distance over the words in each turn.
In this way, using the measures described above we compute the following metrics for each dialog:
Total number of turns;
Average number of words per turn;
Total number of cycles;
Average number of turns in a cycle.
|T1 - A: You know right away what you want.|
|T2 - B: I know right away what we, what we want.|
|T3 - B: I keep hearing about it.|
|T4 - B: I keep hearing the advertisements of it.|
|T5 - A: You can’t find them.|
|T6 - A: You can’t find them.|
|T7 - B: and, i thought i would really miss that.|
|T8 - A: I would, too,|
|Total number of turns: 8|
|Average number of words per turn: 5.9|
|Total number of cycles: 3|
|Average number of turns in cycle: 2|
2.3. Combining Textual and Structural Metrics
The combination of both type of metrics (i.e., textual content and dialog structure) is done using the Borda Count method, by considering the ranking of similarity for a given dialog. In greater detail, for each of the metrics, we first compute the distance matrix , which is an matrix where is total of dialogs, where position contains the distance between dialogs and . This matrix will be computed for the textual content metrics () and for the structural metrics (). Then, for each row in , the ranking matrix , which is also , is computed. Each cell contains the relative ranking of distance compared against all distances , where . We present Tables 4 and 5 to illustrate this process. Considering that in the first row , we assign cells , and with 1, 2 and 3, respectively. The same process is repeated for all the other rows.
That said, for combining the results of two distance matrices denoted and , and their corresponding ranking matrices and , we need to simply compute a third distance matrix denoted , which corresponds to the sum of matrices and , and them normally compute matrix .
In this section, we describe the dataset used for our experiments and present the results of our experimental evaluation and comparison with the two other similarity metrics (e.g., only dialog textual content or only dialog structural features).
We use the Switchboard dataset (SWITCHBOARD92, ) for evaluating our similarity metric. This dataset is composed by 1,154 dialogs, ranging from a minimum of 38 and a maximum of 509 turns. This dataset is one of the most influential spoken corpora, being applied on several different tasks. In this work, we consider the transcripts converted from the spoken dialogs to sequences of text turns.
As described in the previous section, our approach to compute the similarity between two dialogs combines both textual content features and dialog structural features. Since there is no labeled dataset for similar dialogs, we consider an unsupervised evaluation of the metrics.
Our evaluation works as follows. Considering the textual content features and the corresponding ranking matrix , the structure features and the corresponding ranking matrix , and matrix which is the combination of the other two matrices, we compute the mean squared error (MSE) between each matrix with the purpose of understanding how these matrices can be compared. These results are presented in Table 6. We can observe that the MSE between and is much higher compared with that between and or between and . This suggests that our combination method is able to incorporate into the information in both and .
For providing a better idea about the meaning of the computed MSE values, we have computed a random matrix then applied different degrees of perturbation in the ranking, and computed the MSE between the original and perturbed matrices. The perturbation consists of randomly picking a pair of elements and swapping their values. The degree of perturbation corresponds to the number of swaps that are performed. Figure1 plots the MSEs of swapping from , with ranging from 1 to 5. With the values presented in Table 6, we may say that the rankings in and are likely to be very distinct since the MSE between them is above that of swapping 800 pairs. On the other hand, the MSEs of and , and that of and , are between the swapping of 100 and 200 pairs (about 1/5 to 1/3 of the list). Thus, tends to present more similarity to and , compared with the similarity of to .
For a better understanding of the results, we have conducted a qualitative analysis of one dialog. We selected dialog #90 (referred to as Original), and computed the most similar dialogs in terms of Content, which is dialog #940, in terms of Structure, which is dialog #354, and with the combination Content+Structure resulting in dialog #85. Owning to space constraints, we do not present the contents of these dialog, but we present an analysis is terms of metrics for both content and structure.
In Table 8 we list the intersection of the most frequent terms in the dialogs considered as the most similar ones, in terms of Content, Structure, and Content+Structure, in our case study. We can observe that set of terms that intersect the original dialog and the most similar in terms of content, and the set that intersects the original dialog and the most similar in terms of structure, are quite different. Nevertheless, we observe that the terms considering the most similar dialog with Content+Structure cover most of the terms in the other two sets, showing that our combination method can be effective in combining both types of metrics. To complement this analysis, we also show that the intersection of terms among all dialogs is quite small, with only the terms ’people’ and ’think’, which also reinforces the potential of our approach.
|Original and Content||people, course, computer, long, research, problem, think|
|Original and Structure||people, recycling, paper, cans, bottles, plastic, program, location, collected, bins|
|Original and Content+Structure||people, recycling, paper, guy, bins, landfill, computer, plastic, collected, trash, separate, bottles, throw, school, cans|
In Table 9 we present the values found for our structural features, for each of the dialogs. Not surprisingly, we observe that the Original dialog and the most similar one for Structure present very close values. On the other hand, the most similar dialog for Content present the most different values. And the most similar in Content+Structure does not present values as close as that of Structure to the Original dialog, but still the values are quite close if compared with the Content dialog.
|Dialogs||turns||avg words per turn||Cycle||avg turns per cycle|
4. Conclusion and Future Work
In this work, we tackle the problem of dialog similarity. For the best of our knowledge, we are the first to address the problem in the context that uses only the text is not enough to represent similarity between dialogs. We also presented a new metric for dialog similarity based on evidence from both textual content and the dialog structure using the Switchboard dataset. We found that considering structural properties of the dialog, such a number of turns and the present of cycles can improve the similarity detection accuracy when combine of textual content features (e.g., TF-IDF).
As future work, we intend to analyze other temporal properties of the dialog such as precedence actions, other structural features such as speech acts, cycle size and investigate the use of synonyms (e.g., wordvecs) to capture meaning in the similarity. Finally, we would like to analyze the correlation between our similarity metric and human judgments.
-  Antoine Bordes and Jason Weston. Learning end-to-end goal-oriented dialog. arXiv preprint arXiv:1605.07683, 2016.
-  Tamitha Carpenter and Emi Fujioka. The role and identification of dialog acts in online chat. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.
-  Cristian Danescu-Niculescu-Mizil and Lillian Lee. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics, pages 76–87. Association for Computational Linguistics, 2011.
-  Oliver Ferschke, Iryna Gurevych, and Yevgen Chebotar. Behind the article: Recognizing dialog acts in wikipedia talk pages. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 777–786. Association for Computational Linguistics, 2012.
-  J. J. Godfrey, E. C. Holliman, and J. McDaniel. Switchboard: telephone speech corpus for research and development. In [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 517–520 vol.1, Mar 1992.
-  Wael H Gomaa and Aly A Fahmy. A survey of text similarity approaches. International Journal of Computer Applications, 68(13), 2013.
-  Andreas Hotho, Andreas Nürnberger, and Gerhard Paaß. A brief survey of text mining. In Ldv Forum, volume 20, pages 19–62, 2005.
-  Daniel Jurafsky and James H. Martin. Dialog systems and chatbots. In Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, chapter 29, pages 418–440. Prentice Hall PTR, 2000.
-  Faisal M Khan, Todd A Fisher, Lori Shuler, Tianhao Wu, and William M Pottenger. Mining chat-room conversations for social and semantic interactions. Computer Science and Engineering, Lehigh University, 2002.
-  Cemal Köse, Özcan Özyurt, and Guychmyrat Amanmyradov. Mining chat conversations for sex identification. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 45–55. Springer, 2007.
-  Diane J Litman and Kate Forbes-Riley. Predicting student emotions in computer-human tutoring dialogues. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, page 351. Association for Computational Linguistics, 2004.
-  Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023, 2016.
-  Ying Liu, Dengsheng Zhang, Guojun Lu, and Wei-Ying Ma. A survey of content-based image retrieval with high-level semantics. Pattern recognition, 40(1):262–282, 2007.
-  Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. arXiv preprint arXiv:1506.08909, 2015.
-  Beatriz Maeireizo, Diane Litman, and Rebecca Hwa. Co-training for predicting emotions with spoken dialogue data. In Proceedings of the ACL 2004 on Interactive poster and demonstration sessions, page 28. Association for Computational Linguistics, 2004.
-  Hermann A Maurer, Frank Kappe, and Bilal Zaka. Plagiarism-a survey. J. UCS, 12(8):1050–1084, 2006.
Grégoire Mesnil, Yann Dauphin, Kaisheng Yao, Yoshua Bengio, Li Deng, Dilek
Hakkani-Tür, Xiaodong He, Larry Heck, Gokhan Tur, Dong Yu, and Geoffrey
Using recurrent neural networks for slot filling in spoken language understanding.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23:530–539, 2015.
-  Donald Metzler, Susan Dumais, and Christopher Meek. Similarity measures for short segments of text. In European Conference on Information Retrieval, pages 16–27. Springer, 2007.
-  Alan Ritter, Colin Cherry, and Bill Dolan. Unsupervised modeling of twitter conversations. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 172–180. Association for Computational Linguistics, 2010.
-  Dietmar Schabus, Brigitte Krenn, and Friedrich Neubarth. Data-driven identification of dialogue acts in chat messages. Bochumer Linguistische Arbeitsberichte, page 236, 2016.
-  Iulian V. Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. A survey of available corpora for building data-driven dialogue systems. arXiv e-prints, abs/1512.05742, December 2015.
-  Iulian Vlad Serban, Tim Klinger, Gerald Tesauro, Kartik Talamadupula, Bowen Zhou, Yoshua Bengio, and Aaron C Courville. Multiresolution recurrent neural networks: An application to dialogue response generation. In AAAI, pages 3288–3294, 2017.
-  Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. A survey of available corpora for building data-driven dialogue systems. arXiv preprint arXiv:1512.05742, 2015.
-  Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. A hierarchical latent variable encoder-decoder model for generating dialogues. In AAAI, pages 3295–3301, 2017.
-  Ashley Stahl. 10 Ways Enterprise Chatbot Solutions And AI Are Changing The Workplace. https://www.forbes.com/sites/ashleystahl/2017/07/20/10-ways-enterprise-chatbot-solutions-and-ai-are-changing-the-workplace, July 2017.
-  Tomek Strzalkowski, George Aaron Broadwell, Jennifer Stromer-Galley, Samira Shaikh, Sarah Taylor, and Nick Webb. Modeling socio-cultural phenomena in discourse. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 1038–1046. Association for Computational Linguistics, 2010.
-  Pei-Hao Su, David Vandyke, Milica Gasic, Dongho Kim, Nikola Mrksic, Tsung-Hsien Wen, and Steve Young. Learning from real users: Rating dialogue success with neural networks for reinforcement learning in spoken dialogue systems. arXiv preprint arXiv:1508.03386, 2015.
-  Rainer Typke, Frans Wiering, and Remco C Veltkamp. A survey of music information retrieval systems. In Proc. 6th International Conference on Music Information Retrieval, pages 153–160. Queen Mary, University of London, 2005.
-  Marilyn A Walker, Grace I Lin, and Jennifer Sawyer. An annotated corpus of film dialogue for learning and characterizing character style. In LREC, pages 1373–1378, 2012.
-  Sholom M. Weiss, Nitin Indurkhya, and Tong Zhang. Fundamentals of Predictive Text Mining. Springer Publishing Company, Incorporated, 2012.
-  Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv:1604.04562, 2016.
-  Jason D Williams, Kavosh Asadi, and Geoffrey Zweig. Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning. arXiv preprint arXiv:1702.03274, 2017.
-  Jason D Williams and Geoffrey Zweig. End-to-end lstm-based dialog control optimized with supervised and reinforcement learning. arXiv preprint arXiv:1606.01269, 2016.