Task-oriented dialogue models aim at assisting with well-defined and structured problems like booking tickets or providing information to visitors in a new city Raux et al. (2005). Most current industry-oriented systems rely on modular, domain-focused frameworks Young et al. (2013); Sarikaya et al. (2016), with separate components for user understanding Henderson et al. (2014), decision making Gašić et al. (2010) and system answer generation Wen et al. (2015). Recent progress in sequence-to-sequence (seq2seq) modelling has enabled the development of fully neural end-to-end architectures, allowing for different components to be optimized jointly in order to share information Wen et al. (2017); Zhao et al. (2017); Budzianowski and Vulić (2019).
Dialogue systems benefit greatly from optimizing on detailed annotations, such as turn-level dialogue state labels Henderson et al. (2014) or dialogue actions Rieser and Lemon (2011), with end-to-end architectures still relying on intermediate labels in order to obtain satisfactory results Lei et al. (2018). Collecting these labels is often the bottleneck in dataset creation, as the process is expensive and time-consuming, requiring domain and expert knowledge Asri et al. (2017). Due to this restriction, existing datasets for task-oriented dialogue are several orders of magnitude smaller compared to general open-domain dialogue corpora Lowe et al. (2015); Henderson et al. (2019).
Arguably, one of the most challenging parts of dialogue modelling is maintaining an interpretable internal memory over crucial domain concepts Williams et al. (2016)
. Although there is increasing research effort to learn Dialogue State Tracking (DST) jointly with the text generation componentEric et al. (2017); Wu et al. (2019), the most effective models use it as an intermediate signal Wen et al. (2017); Lei et al. (2018). The difficulty of state tracking has made this task a driving force behind most of the Dialog System Technology Challenges in recent years Henderson et al. (2014); Kim et al. (2017).
In this paper, we reduce the reliance of task-oriented dialogue systems on data collection by leveraging semi-supervised training Chapelle et al. (2009). Two approaches are investigated and evaluated for providing an improved training signal to the dialogue state tracking component in an end-to-end dialogue system. Automatically predicted DST output on unlabelled utterances is treated as additional annotation if the model confidence is sufficiently high. Furthermore, subtle perturbations of existing datapoints are created, optimizing for their predictions to be similar to the original instances. Our analysis on the MultiWOZ corpus Budzianowski et al. (2018), covering a range of domains and topics, finds that these methods can reduce intermediate annotation by up to 30% while maintaining equivalent system performance. We also describe and evaluate the first end-to-end dialogue model created for the MultiWOZ corpus.
2 End-to-end Neural Dialogue Model
We now present the end-to-end neural dialogue model composed of three main components: dialogue state tracker, policy network and natural language generator. These components are trained jointly as a connected network and will be introduced in detail in the next paragraphs. The overall architecture can be seen in Figure 1.
Dialogue State Tracker (DST)
The DST is responsible for both understanding the input user utterance and updating the internal dialogue state for the downstream components. There are two types of slots which can be detected in the input: informable slots and requestable slots. The former describes the attributes of the entity that the user is looking for, e.g., pricerange of a hotel. The latter captures information that the user desires to know about the entity, e.g., postcode of a hotel. Each informable slot contains several possible values with two special labels: not-mentioned and don’t-care. A good DST should be able to correctly recognize the mentioned slot-value pairs in the user utterance and to maintain the updated dialogue (belief) state.
Let , and denote the index of domain, slot and value. As depicted at the top of Figure 1, the user utterance : at turn t is first encoded by the BiLSTM to obtain the hidden states . The encoding of the slot-value pair
is the output of the affine layer that takes the concatenation of the embeddings of domain, slot and value
as the input. The context vectoris then computed by the attention mechanism, denoted as attn in Figure 1, following Luong et al. (2015):
where is the word index of the user utterance and sim denotes any function that calculates the similarity of two vectors. We adopt here the dot product function, following Mrkšić et al. (2017); Zhong et al. (2018); Ramadan et al. (2018). The similarity score between and is then computed to see whether the slot-value pair
is mentioned in the utterance. The mentioned pair should have higher similarity score to its context vector than those which are not mentioned. The softmax layer is then applied to form the probability distributionfor each informable slot , where the predicted value is the value with the highest probability. The same attention mechanism is used for each requestable slot to decide whether the user has asked for the slot in the current turn. The sigmoid layer is used instead as it is a binary classification problem. The prediction of requestable slots will be used as input to the natural language generator.
The belief state is the concatenation of the distributions over all informable slot-value pairs that is updated at each turn to keep track of the information provided by the user during the entire conversation. To form the belief state at turn , for each informable slot the update mechanism checks if the predicted value is either not-mention or dont-care for the current turn. If it is, then the probabilistic distribution in is kept, otherwise it is updated by the new distribution at the current turn.
The policy network is responsible for fusing the signals from the belief state , the encoding of the user utterance and the database query result . The database query is constructed from the predicted belief state. The number of all entities that match the predictions of the DST form the database query vector111Following Wen et al. (2017), by querying the database using as query, is the 1-hot representation with each element indicating different number of the matching entities in the database. We use 5 bins to indicate the matching number from 0 to 3 and more than 3.. We use a simple feedforward layer as the policy network:
where [*] denotes the concatenation of vectors.
Natural Language Generator
Taking the input from the policy network and predictions of requestable slots from the tracker, the generator outputs a system response word-by-word until the <EOS> token is generated. To improve the generation of correct slots corresponding to the user input, we adopt the semantically-conditioned LSTM Wen et al. (2015) that contains a self-updating gate memory to record the produced slots in the generated sentence.
The model is optimized jointly against two sources of information – DST intermediate labels and system utterances. The DST loss consists of the cross-entropy over the multi-class classification of informable slots while binary cross-entropy is used for requestable slots:
where , and are the index of domain, informable slot and requestable slot respectively; is the target distribution. The generation error is a standard cross-entropy between the predicted words and target words:
where is the word index in the generated sentence. To jointly train the DST, the policy network, and the generator as a connected network, the final objective function becomes .
The DST loss requires each turn to be manually annotated with the correct slot-value pairs. We experiment with two different semi-supervised training methods that take advantage of unlabelled examples instead, allowing us to reduce the amount of required annotation. The first approach is based on the pseudo-labelling strategy by Chapelle et al. (2009). If the prediction probability of an unlabelled data point for a particular class is larger than a given threshold , the example is included in the DST loss with the predicted label. The parameter is optimized on the validation set during development.
The second semi-supervised technique investigated is the -model Sajjadi et al. (2016) where the input is perturbed with random noise . The perturbations are applied to both labelled and unlabelled data points at the level of embedding of user utterance. The model is then required to produce similar predictions over the belief state compared to the original input, optimized with an additional loss:
where is the batch size and
is a hyperparameter controlling the weight of the loss.
We investigate the effects of semi-supervised training on optimizing the end-to-end neural dialogue system. In particular, we evaluate how much annotation at the intermediate-level could be reduced while preserving comparable results of the overall dialogue task completion metrics.
The three analyzed models are evaluated on the MultiWOZ dataset consisting of 10,438 task-oriented dialogues Budzianowski et al. (2018). The conversations in MultiWOZ are natural as they were gathered based on human-to-human interactions following the Wizard-of-Oz paradigm. The corpus includes multi-domain conversations spanning across domains including Restaurant, Hotel, Attraction, Train and Taxi. The size of the dataset allows us to control the amount of available fully labelled datapoints.
There are two metrics of importance when evaluating task-oriented dialogue systems. The first is the DST joint goal accuracy, defined as an average joint accuracy over all slots per turn Williams et al. (2016). The second is the Success metric that informs how many times systems have presented the entity satisfying the user’s goal and provided them with all the additional requested information Wen et al. (2017). The models are optimized using the validation set and the results are averaged over 10 different seeds.
Varying data amount
We examine the performance of the baseline model compared to the two semi-supervised models as the amount of labelled data varies. The result of the DST joint accuracy is presented in Figure 2. The pseudo-labelling model performs better than the baseline when more than 50% of the dataset is labelled. At the scarce data levels (10% and 30%) , the pseudo-labelling model is not producing pseudo training points that help improve DST predictions. In contrast, the -model takes advantages of the additional regularization loss and effectively leverages unlabelled data to enhance the performance over the baseline. The improvements are consistently more than 5% when training with 30 to 90% of labelled data and even reach the performance of the fully trained baseline model with only 70% labelled data.
Figure 3 shows the Success metric results. The pseudo-labelling method is not able to improve performance over the baseline regardless of the amount of labelled data. However, the -model is capable of improving the success rate consistently and manages to reach the performance of the fully trained model with only 50% of the intermediate DST signal. Note that a better DST joint accuracy does not necessarily translate to a better success rate as the final metric is also influenced by the quality of the generator.
DST joint accuracy considers all slot-value pairs in an utterance and cannot give us further insight regarding the source of the improvements. We are particularly interested in whether the semi-supervised models can leverage unlabelled data to improve the prediction of rarely seen slot-value pairs. In this analysis, we classify all slot-value pairs in the test set in terms of their number of training examples in 50% of the labelled data. Table1 presents the results, showing that the -model improves accuracy by 5% when the slot-value pair is rarely (1-10 times) seen during training. The improvement on few-shot slots contributes to the improvement of joint accuracy.
We also investigate if the improvements in the success rate are consistent among all domains. Figure 4 shows the success rate on individual domains in the case of 50% of data is labelled. Both semi-supervised models improve performance over the baseline in all domains except for the taxi domain. We hypothesize this comes from the fact that the taxi domain is a relatively easy domain with only possible slots.
4 Conclusions and Future Work
In this paper, we have analyzed how much semi- supervised techniques could help to reduce the need for intermediate-level annotations in training neural task-oriented dialogue models. The results suggest that we do not need to annotate all intermediate signals and are able to leverage unannotated examples for training these components instead. In the future, we plan to experiment with other intermediate signals like dialogue acts. Further improvements could potentially be obtained from employing more advanced regularization losses Oliver et al. (2018).
Bo-Hsiang Tseng is supported by Cambridge Trust and the Ministry of Education, Taiwan. Marek Rei’s research is supported by Cambridge English via the ALTA Institute. Paweł Budzianowski is funded by an EPSRC grant (EP/M018946/1) and by Toshiba Research Europe Ltd, Cambridge Research Laboratory (RG85875).
- Frames: a corpus for adding memory to goal-oriented dialogue systems. Proceedings of SigDial, pp. 207–219. External Links: Cited by: §1.
- Hello, it’s GPT-2 - how can I help you? towards the use of pretrained language models for task-oriented dialogue systems. arXiv preprint arXiv:1907.05774. External Links: Cited by: §1.
- MultiWOZ-A Large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of EMNLP, pp. 5016–5026. External Links: Cited by: §1, §3.
Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews].
IEEE Transactions on Neural Networks20 (3), pp. 542–542. External Links: Cited by: §1, §2.
- Key-value retrieval networks for task-oriented dialogue. In Proceedings of SigDial, pp. 37–49. External Links: Cited by: §1.
- Gaussian processes for fast policy optimisation of pomdp-based dialogue managers. In Proceedings of SigDial, pp. 201–204. External Links: Cited by: §1.
- A repository of conversational datasets. arXiv preprint arXiv:1904.06472. Cited by: §1.
- The third dialog state tracking challenge. In Spoken Language Technology Workshop (SLT), 2014 IEEE, pp. 324–329. External Links: Cited by: §1, §1, §1.
- The fourth dialog state tracking challenge. In Dialogues with Social Robots, pp. 435–449. External Links: Cited by: §1.
- Sequicity: simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In Proceedings of ACL, pp. 1437–1447. External Links: Cited by: §1, §1.
- The ubuntu dialogue corpus: a large dataset for research in unstructured multi-turn dialogue systems. In 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 285. External Links: Cited by: §1.
Effective approaches to attention-based neural machine translation. In
Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412–1421. External Links: Cited by: §2.
- Neural belief tracker: data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1, pp. 1777–1788. External Links: Cited by: §2.
- Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, pp. 3235–3246. External Links: Cited by: §4.
- Large-scale multi-domain belief tracking with knowledge sharing. In Proceedings of ACL, pp. 432–437. External Links: Cited by: §2.
- Let’s go public! taking a spoken dialog system to the real world. In Ninth European Conference on Speech Communication and Technology, External Links: Cited by: §1.
- Reinforcement learning for adaptive dialogue systems: a data-driven methodology for dialogue management and natural language generation. Springer Science & Business Media. External Links: Cited by: §1.
- Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems, pp. 1163–1171. External Links: Cited by: §2.
- An overview of end-to-end language understanding and dialog management for personal digital assistants. In 2016 IEEE Spoken Language Technology Workshop (SLT), pp. 391–397. External Links: Cited by: §1.
- Semantically conditioned lstm-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Cited by: §1, §2.
- A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of EACL, pp. 438–449. External Links: Cited by: §1, §1, §3, footnote 1.
- The dialog state tracking challenge series: a review. Dialogue & Discourse 7 (3), pp. 4–33. External Links: Cited by: §1, §3.
- Global-to-local memory pointer networks for task-oriented dialogue. ICLR. External Links: Cited by: §1.
- POMDP-based statistical spoken dialog systems: A review. Proceedings of the IEEE 101 (5), pp. 1160–1179. External Links: Cited by: §1.
Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of ACL, pp. 654–664. External Links: Cited by: §1.
- Global-locally self-attentive encoder for dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1458–1467. External Links: Cited by: §2.