Chat-Bot-Kit: A web-based tool to simulate text-based interactions between humans and with computers

11/02/2019 ∙ by Kyoko Sugisaki, et al. ∙ 0

In this paper, we describe Chat-Bot-Kit, a web-based tool for text-based chats that we designed for research purposes in computer-mediated communication (CMC). Chat-Bot-Kit enables to carry out language studies on text-based real-time chats for the purpose of research: The generated messages are structured with language performance data such as pause and speed of keyboard-handling and the movement of the mouse. The tool provides two modes of chat communications - quasi-synchron and synchron modes - and various typing indicators. The tool is also designed to be used in wizard-of-oz studies in Human-Computer Interaction (HCI) and for the evaluation of chatbots (dialogue systems) in Natural Language Processing (NLP).



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Chat tool

Text messaging is a very popular form of communication, and apps like Facebook Messenger, SnapChat, WeChat, or WhatsApp belong to the most installed apps worldwide. Text messaging is, in recent years, a hotbet for technical and user experience innovations: Small changes of rules and design lead to completely different usage patterns, usage scenarios and overall user experiences. As an example, Snapchat lets the messages disappear a short time after the are consumed/read. Just by doing so, messaging becomes a more ephemeral form of communication that does not leave traces or tangible artefacts behind and is providing a safer and trusted space to exchange more sensitive and private topics. WhatsApp lets users see the status of messages, i.e. whether they are transmitted successfully to the recipient or read, and it shows the online activity of others, which renders the interaction again more closely to face-to-face communication, compared to classical text messaging services such as SMS.

In recent years, major tech companies have also started to offer chatbot platforms that allow businesses to automate conversations with consumers and reach them where they spend a lot of their time (i.e. within popular messaging apps). On Facebook Messenger alone, more than 300.000 chatbots have been deployed by Mid 2018. Many industry observers claim that chatbots development will replace app development. At the same time, consumers struggle with the interactions (Brandtzæg and Følstad, 2018; Følstad and Brandtzæg, 2017). At the current state of the technology, the text-based chat communication with computers is needed to be better understood, more actively designed and more frequently tested to make them more usable and valuable for users, let alone feel more natural. A common way to simulate conversations with chatbots in a cheap, versatile way without actually implementing them is to conduct so called Wizard of Oz studies (woz), in which a researcher pretends to be a computer during a conversation with a human test subject who is briefed to talk with a computer. Such studies are common in the field of CMC, Natural Language Processing (NLP) and Human-Computer-Interaction.

Figure 1. Chat-Bot-Kit

There are a number of tools with which one could simulate chats and WoZ-studies. One could use a tool like Skype, for example. There are also some commercial chat tools that provide a log file that is useful for the purpose of research (e.g., (Medhi Thies et al., 2017)). However, there are some missing features for the purpose of research.

Chat-Bot-Kit, a web-based chat tool (Fig. 1) that we present in this paper has been developed based on the following requirements for research in Computer-mediated communication (CMC), Human-Computer Interaction (HCI) and Natural Language Processing (NLP).

  • The submitted messages can be exported into a structured data format (Microsoft Office Excel) for the purpose of further analysis on the chat communication.

  • The language performance data is embedded into the structure of messages. The tool automatically measures the pause, speed, rhythm of keyboard stokes and the movement of the mouse, next to the time stamp of the message submission. The temporal feature of the chat interaction is of relevance in CMC (cf. (Jones, 2013))

  • A user can have more than one name and role: this is considered to be used in HCI. In case of a wizard-of-oz simulation study (Li et al., 2007; Dow et al., 2005; Dahlbäck et al., 1993), the identity of chat constructors (for example, an agent of an insurance company) plays an important role for the credibility of wizards as machine.

  • The submitted messages can be edited, rated and commented directly in the user interface and the data is integrated into the output file. The feature is considered for HCI and NLP studies in which the designer or developer of a chatbot wants to test the system and carry out the user evaluation on the level of messages.

  • The tool is scalable: the system is based on AngularJS components that are easy to be extended to a wizard assistant tool or a chatbot

  • There are several options of the chat communication with regard to the turn-taking.

    • Typing indicator: the typing indicator (Gnewuch et al., 2018) displays the typing behaviour of the communication partner. This feature affects the turn-taking, as it is a type of mutual monitoring between communicators. In our tool, the typing indicator can be configured by the communicators and study leaders.

    • Quasi-synchronous and synonymous mode: In the CMC, the text-chat communication is currently investigated in the quasi-synchronous mode. The quasi-synchronous mode has dominated text-based chats (such as WhatsApp or Skype), thus has been investigated for a long time (e.g., (Garcia and Jacobs, 1999; Herring, 1999)). While quasi-synchronous written communication is regarded as one-way transmission of turns and the on-going typing is only visible to the writer, but not to the other participants, synchronous written communication is characterised in two-way transmission where the participants transmit their typing in a keystroke-by-keystoke to their own window, instead of into one window for all interlocutors in the quasi-synchronous one. In the past, there were frameworks such UNIX talk or VAX phone for the synchronous one (cf. (Herring, 1999)). Google Wave was also experimenting with synchronous modes. The synchronicity of the communication is one crucial difference in face-to-face/phones and CMS that affects the interaction deeply, in particular in turn-taking (Garcia and Jacobs, 1999; Herring, 1999) and repair (Jacobs, 2013). The tool provides these two modes of CMC, ready to be used in studies.

2. Conclusion

In this paper, we presented our web-based chat tool designed for the research in Computer-mediated communication (CMC), Human-Computer Interaction (HCI) and Natural Language Processing (NLP). In future work, we plan to extend several wizard assistance methods for wizard-of-oz studies in HCI that include machine learning and allow to iteratively train models during a study.


  • P. B. Brandtzæg and A. Følstad (2018) Chatbots: changing user needs and motivations. Interactions 25 (5), pp. 38–43. Cited by: §1.
  • N. Dahlbäck, A. Jönsson, and L. Ahrenberg (1993) Wizard of Oz studies — why and how. Knowledge-Based Systems 6 (4), pp. 258 – 266. Cited by: 3rd item.
  • S. Dow, B. MacIntyre, J. Lee, C. Oezbek, J. D. Bolter, and M. Gandy (2005) Wizard of oz support throughout an iterative design process. IEEE Pervasive Computing 4 (4), pp. 18–26. Cited by: 3rd item.
  • A. Følstad and P. B. Brandtzæg (2017) Chatbots and the new world of hci. Interactions 24 (4), pp. 38–42. Cited by: §1.
  • A. C. Garcia and J. B. Jacobs (1999) The eyes of the beholder: understanding the turn-taking system in quasi-synchronous computer-mediated communication. Research on Language and Social Interaction 32 (4), pp. 337–367. Cited by: 2nd item.
  • U. Gnewuch, S. Morana, M. Thomas, P. Adam, and A. Maedche (2018) “The chatbot is typing …” – the role of typing indicators in human-chatbot interaction. Cited by: 1st item.
  • S. Herring (1999) Interactional Coherence in CMC. Journal of Computer-Mediated Communication 4 (4). Cited by: 2nd item.
  • G. A. C. Jacobs (2013) Repair in chat room interaction. In Pragmatics of computer-mediated communication, S. C. Herring (Ed.), Cited by: 2nd item.
  • R. H. Jones (2013) Rhythm and timing in chat room interaction. In Pragmatics of computer-mediated communication, S. C. Herring (Ed.), Cited by: 2nd item.
  • Y. Li, J. I. Hong, and J. A. Landay (2007) Design challenges and principles for wizard of oz testing of location-enhanced applications. IEEE Pervasive Computing 6 (2), pp. 70–75. Cited by: 3rd item.
  • I. Medhi Thies, N. Menon, S. Magapu, M. Subramony, and J. O’Neill (2017) How do you want your chatbot? an exploratory wizard-of-oz study with young, urban indians. In Human-Computer Interaction (INTERACT 2017), R. Bernhaupt, G. Dalvi, A. Joshi, D. K. Balkrishan, J. O’Neill, and M. Winckler (Eds.), Cham, pp. 441–459. Cited by: §1.