PyTorch implementation for Interpretable Dialog Generation, It is released by Tiancheng Zhao (Tony) from Dialog Research Center, LTI, CMU
The encoder-decoder dialog model is one of the most prominent methods used to build dialog systems in complex domains. Yet it is limited because it cannot output interpretable actions as in traditional systems, which hinders humans from understanding its generation process. We present an unsupervised discrete sentence representation learning method that can integrate with any existing encoder-decoder dialog models for interpretable response generation. Building upon variational autoencoders (VAEs), we present two novel models, DI-VAE and DI-VST that improve VAEs and can discover interpretable semantics via either auto encoding or context predicting. Our methods have been validated on real-world dialog datasets to discover semantic representations and enhance encoder-decoder models with interpretable generation.READ FULL TEXT VIEW PDF
The encoder-decoder models for unsupervised sentence representation lear...
Autoencoders are a deep learning model for representation learning. When...
Context information plays an important role in human language understand...
Recently several deep learning based models have been proposed for end-t...
This paper examines various unsupervised pretraining objectives for lear...
Learning a shared dialog structure from a set of task-oriented dialogs i...
While recent neural encoder-decoder models have shown great promise in
PyTorch implementation for Interpretable Dialog Generation, It is released by Tiancheng Zhao (Tony) from Dialog Research Center, LTI, CMU
Classic dialog systems rely on developing a meaning representation to represent the utterances from both the machine and human users Larsson and Traum (2000); Bohus et al. (2007). The dialog manager of a conventional dialog system outputs the system’s next action in a semantic frame that usually contains hand-crafted dialog acts and slot values Williams and Young (2007)
. Then a natural language generation module is used to generate the system’s output in natural language based on the given semantic frame. This approach suffers from generalization to more complex domains because it soon become intractable to manually design a frame representation that covers all of the fine-grained system actions. The recently developed neural dialog system is one of the most prominent frameworks for developing dialog agents in complex domains. The basic model is based on encoder-decoder networksCho et al. (2014) and can learn to generate system responses without the need for hand-crafted meaning representations and other annotations.
Although generative dialog models have advanced rapidly Serban et al. (2016); Li et al. (2016); Zhao et al. (2017), they cannot provide interpretable system actions as in the conventional dialog systems. This inability limits the effectiveness of generative dialog models in several ways. First, having interpretable system actions enables human to understand the behavior of a dialog system and better interpret the system intentions. Also, modeling the high-level decision-making policy in dialogs enables useful generalization and data-efficient domain adaptation Gašić et al. (2010). Therefore, the motivation of this paper is to develop an unsupervised neural recognition model that can discover interpretable meaning representations of utterances (denoted as latent actions) as a set of discrete latent variables from a large unlabelled corpus as shown in Figure 1. The discovered meaning representations will then be integrated with encoder decoder networks to achieve interpretable dialog generation while preserving all the merit of neural dialog systems.
We focus on learning discrete latent representations instead of dense continuous ones because discrete variables are easier to interpret van den Oord et al. (2017)
and can naturally correspond to categories in natural languages, e.g. topics, dialog acts and etc. Despite the difficulty of learning discrete latent variables in neural networks, the recently proposed Gumbel-Softmax offers a reliable way to back-propagate through discrete variablesMaddison et al. (2016); Jang et al. (2016). However, we found a simple combination of sentence variational autoencoders (VAEs) Bowman et al. (2015) and Gumbel-Softmax fails to learn meaningful discrete representations. We then highlight the anti-information limitation of the evidence lowerbound objective (ELBO) in VAEs and improve it by proposing Discrete Information VAE (DI-VAE) that maximizes the mutual information between data and latent actions. We further enrich the learning signals beyond auto encoding by extending Skip Thought Kiros et al. (2015) to Discrete Information Variational Skip Thought (DI-VST) that learns sentence-level distributional semantics. Finally, an integration mechanism is presented that combines the learned latent actions with encoder decoder models.
The proposed systems are tested on several real-world dialog datasets. Experiments show that the proposed methods significantly outperform the standard VAEs and can discover meaningful latent actions from these datasets. Also, experiments confirm the effectiveness of the proposed integration mechanism and show that the learned latent actions can control the sentence-level attributes of the generated responses and provide human-interpretable meaning representations.
Our work is closely related to research in latent variable dialog models. The majority of models are based on Conditional Variational Autoencoders (CVAEs) Serban et al. (2016); Cao and Clark (2017) with continuous latent variables to better model the response distribution and encourage diverse responses. Zhao et al., Zhao et al. (2017) further introduced dialog acts to guide the learning of the CVAEs. Discrete latent variables have also been used for task-oriented dialog systems Wen et al. (2017), where the latent space is used to represent intention. The second line of related work is enriching the dialog context encoder with more fine-grained information than the dialog history. Li et al., Li et al. (2016) captured speakers’ characteristics by encoding background information and speaking style into the distributed embeddings. Xing et al., Xing et al. (2016) maintain topic encoding based on Latent Dirichlet Allocation (LDA) Blei et al. (2003) of the conversation to encourage the model to output more topic coherent responses.
The proposed method also relates to sentence representation learning using neural networks. Most work learns continuous distributed representations of sentences from various learning signalsHill et al. (2016), e.g. the Skip Thought learns representations by predicting the previous and next sentences Kiros et al. (2015). Another area of work focused on learning regularized continuous sentence representation, which enables sentence generation by sampling the latent space Bowman et al. (2015); Kim et al. (2017). There is less work on discrete sentence representations due to the difficulty of passing gradients through discrete outputs. The recently developed Gumbel Softmax Jang et al. (2016); Maddison et al. (2016)
and vector quantizationvan den Oord et al. (2017) enable us to train discrete variables. Notably, discrete variable models have been proposed to discover document topics Miao et al. (2016) and semi-supervised sequence transaction Zhou and Neubig (2017)
Our work differs from these as follows: (1) we focus on learning interpretable variables; in prior research the semantics of latent variables are mostly ignored in the dialog generation setting. (2) we improve the learning objective for discrete VAEs and overcome the well-known posterior collapsing issue Bowman et al. (2015); Chen et al. (2016)
. (3) we focus on unsupervised learning of salient features in dialog responses instead of hand-crafted features.
Our formulation contains three random variables: the dialog context, the response and the latent action . The context often contains the discourse history in the format of a list of utterances. The response is an utterance that contains a list of word tokens. The latent action is a set of discrete variables that define high-level attributes of . Before introducing the proposed framework, we first identify two key properties that are essential in order for to be interpretable:
should capture salient sentence-level features about the response .
The meaning of latent symbols should be independent of the context .
The first property is self-evident. The second can be explained: assume contains a single discrete variable with classes. Since the context can be any dialog history, if the meaning of each class changes given a different context, then it is difficult to extract an intuitive interpretation by only looking at all responses with class . Therefore, the second property looks for latent actions that have context-independent semantics so that each assignment of conveys the same meaning in all dialog contexts.
With the above definition of interpretable latent actions, we first introduce a recognition network and a generation network . The role of is to map an sentence to the latent variable and the generator defines the learning signals that will be used to train ’s representation. Notably, our recognition network does not depend on the context as has been the case in prior work Serban et al. (2016). The motivation of this design is to encourage to capture context-independent semantics, which are further elaborated in Section 3.4. With the learned by and , we then introduce an encoder decoder network and and a policy network . At test time, given a context , the policy network and encoder decoder will work together to generate the next response via . In short, , , and are the four components that comprise our proposed framework. The next section will first focus on developing and for learning interpretable and then will move on to integrating with and in Section 3.3.
Our baseline model is a sentence VAE with discrete latent space. We use an RNN as the recognition network to encode the response . Its last hidden state is used to represent . We define
to be a set of K-way categorical variables, where is the number of variables. For each , its posterior distribution is defined as
. During training, we use the Gumbel-Softmax trick to sample from this distribution and obtain low-variance gradients. To map the latent samples to the initial state of the decoder RNN, we definewhere and is the generator cell size. Thus the initial state of the generator is: . Finally, the generator RNN is used to reconstruct the response given . VAEs is trained to maxmimize the evidence lowerbound objective (ELBO) Kingma and Welling (2013). For simplicity, later discussion drops the subscript in and assumes a single latent . Since each is independent, we can easily extend the results below to multiple variables.
It is well-known that sentence VAEs are hard to train because of the posterior collapse issue. Many empirical solutions have been proposed: weakening the decoder, adding auxiliary loss etc. Bowman et al. (2015); Chen et al. (2016); Zhao et al. (2017). We argue that the posterior collapse issue lies in ELBO and we offer a novel decomposition to understand its behavior. First, instead of writing ELBO for a single data point, we write it as an expectation over a dataset:
where and is the mutual information between and . This expansion shows that the KL term in ELBO is trying to reduce the mutual information between latent variables and the input data, which explains why VAEs often ignore the latent variable, especially when equipped with powerful decoders.
A natural solution to correct the anti-information issue in Eq. 3 is to maximize both the data likelihood lowerbound and the mutual information between and the input data:
Therefore, jointly optimizing ELBO and mutual information simply cancels out the information-discouraging term. Also, we can still sample from the prior distribution for generation because of . Eq. 4 is similar to the objectives used in adversarial autoencoders Makhzani et al. (2015); Kim et al. (2017). Our derivation provides a theoretical justification to their superior performance. Notably, Eq. 4
arrives at the same loss function proposed in infoVAEZhao S et al. (2017). However, our derivation is different, offering a new way to understand ELBO behavior.
The remaining challenge is how to minimize , since is an expectation over . When is continuous, prior work has used adversarial training Makhzani et al. (2015); Kim et al. (2017) or Maximum Mean Discrepancy (MMD) Zhao S et al. (2017) to regularize . It turns out that minimizing for discrete is much simpler than its continuous counterparts. Let be a sample from a batch of data points. Then we have:
where is a mixture of softmax from the posteriors of each . We can approximate by:
We refer to Eq. 6 as Batch Prior Regularization (BPR). When approaches infinity, approaches the true marginal distribution of . In practice, we only need to use the data from each mini-batch assuming that the mini batches are randomized. Last, BPR is fundamentally different from multiplying a coefficient to anneal the KL term in VAE Bowman et al. (2015). This is because BPR is a non-linear operation log_sum_exp. For later discussion, we denote our discrete infoVAE with BPR as DI-VAE.
DI-VAE infers sentence representations by reconstruction of the input sentence. Past research in distributional semantics has suggested the meaning of language can be inferred from the adjacent context Harris (1954); Hill et al. (2016). The distributional hypothesis is especially applicable to dialog since the utterance meaning is highly contextual. For example, the dialog act is a well-known utterance feature and depends on dialog state Austin (1975); Stolcke et al. (2000). Thus, we introduce a second type of latent action based on sentence-level distributional semantics.
Skip thought (ST) is a powerful sentence representation that captures contextual information Kiros et al. (2015). ST uses an RNN to encode a sentence, and then uses the resulting sentence representation to predict the previous and next sentences. Inspired by ST’s robust performance across multiple tasks Hill et al. (2016), we adapt our DI-VAE to Discrete Information Variational Skip Thought (DI-VST) to learn discrete latent actions that model distributional semantics of sentences. We use the same recognition network from DI-VAE to output ’s posterior distribution . Given the samples from , two RNN generators are used to predict the previous sentence and the next sentences . Finally, the learning objective is to maximize:
We now describe how to integrate a given with an encoder decoder and a policy network. Let the dialog context be a sequence of utterances. Then a dialog context encoder network can encode the dialog context into a distributed representation . The decoder can generate the responses using samples from . Meanwhile, we train to predict the aggregated posterior from via maximum likelihood training. This model is referred as Latent Action Encoder Decoder (LAED) with the following objective.
Also simply augmenting the inputs of the decoders with latent action does not guarantee that the generated response exhibits the attributes of the give action. Thus we use the controllable text generation framework Hu et al. (2017) by introducing , which reuses the same recognition network as a fixed discriminator to penalize the decoder if its generated responses do not reflect the attributes in .
Since it is not possible to propagate gradients through the discrete outputs at at each word step, we use a deterministic continuous relaxation Hu et al. (2017) by replacing output of
with the probability of each word. Letbe the normalized probability at step , the inputs to at time are then the sum of word embeddings weighted by , i.e. and is the word embedding matrix. Finally this loss is combined with
and a hyperparameterto have Attribute Forcing LAED.
Despite their similarities, we highlight the key differences that prohibit CVAE from achieving interpretable dialog generation. First encourages Agakov (2005), which learns that capture context-dependent semantics. More intuitively, in CVAE is trained to generate via so the meaning of learned can only be interpreted along with its context . Therefore this violates our goal of learning context-independent semantics. Our methods learn that only depends on and trains separately to ensure the semantics of are interpretable standalone.
The proposed methods are evaluated on four datasets. The first corpus is Penn Treebank (PTB) Marcus et al. (1993) used to evaluate sentence VAEs Bowman et al. (2015). We used the version pre-processed by Mikolov Mikolov et al. (2010). The second dataset is the Stanford Multi-Domain Dialog (SMD) dataset that contains 3,031 human-Woz, task-oriented dialogs collected from 3 different domains (navigation, weather and scheduling) Eric and Manning (2017). The other two datasets are chat-oriented data: Daily Dialog (DD) and Switchboard (SW) Godfrey and Holliman (1997), which are used to test whether our methods can generalize beyond task-oriented dialogs but also to to open-domain chatting. DD contains 13,118 multi-turn human-human dialogs annotated with dialog acts and emotions. Li et al. (2017). SW has 2,400 human-human telephone conversations that are annotated with topics and dialog acts. SW is a more challenging dataset because it is transcribed from speech which contains complex spoken language phenomenon, e.g. hesitation, self-repair etc.
The first experiment used PTB and DD to evaluate the performance of the proposed methods in learning discrete sentence representations. We implemented DI-VAE and DI-VST using GRU-RNN Chung et al. (2014) and trained them using Adam Kingma and Ba (2014). Besides the proposed methods, the following baselines are compared. Unregularized models: removing the KL term from DI-VAE and DI-VST leads to a simple discrete autoencoder (DAE) and discrete skip thought (DST) with stochastic discrete hidden units. ELBO models: the basic discrete sentence VAE (DVAE) or variational skip thought (DVST) that optimizes ELBO with regularization term KL. We found that standard training failed to learn informative latent actions for either DVAE or DVST because of the posterior collapse. Therefore, KL-annealing Bowman et al. (2015) and bag-of-word loss Zhao et al. (2017) are used to force these two models learn meaningful representations. We also include the results for VAE with continuous latent variables reported on the same PTB Zhao et al. (2017). Additionally, we report the perplexity from a standard GRU-RNN language model Zaremba et al. (2014).
The evaluation metrics include reconstruction perplexity (PPL), KLand the mutual information between input data and latent variables . Intuitively a good model should achieve low perplexity and KL distance, and simultaneously achieve high . The discrete latent space for all models are =20 and =10. Mini-batch size is 30.
Table 1 shows that all models achieve better perplexity than an RNNLM, which shows they manage to learn meaningful . First, for auto-encoding models, DI-VAE is able to achieve the best results in all metrics compared other methods. We found DAEs quickly learn to reconstruct the input but they are prone to overfitting during training, which leads to lower performance on the test data compared to DI-VAE. Also, since there is no regularization term in the latent space, is very different from the
which prohibits us from generating sentences from the latent space. In fact, DI-VAE enjoys the same linear interpolation properties reported inBowman et al. (2015) (See Appendix A.2). As for DVAEs, it achieves zero in standard training and only manages to learn some information when training with KL-annealing and bag-of-word loss. On the other hand, our methods achieve robust performance without the need for additional processing. Similarly, the proposed DI-VST is able to achieve the lowest PPL and similar KL compared to the strongly regularized DVST. Interestingly, although DST is able to achieve the highest , but PPL is not further improved. These results confirm the effectiveness of the proposed BPR in terms of regularizing while learning meaningful posterior .
In order to understand BPR’s sensitivity to batch size , a follow-up experiment varied the batch size from 2 to 60 (If =1, DI-VAE is equivalent to DVAE).
Figure 2 show that as increases, perplexity, monotonically improves, while only increases from 0 to 0.159. After , the performance plateaus. Therefore, using mini-batch is an efficient trade-off between estimation and computation speed.
The last experiment in this section investigates the relation between representation learning and the dimension of the latent space. We set a fixed budget by restricting the maximum number of modes to be about 1000, i.e. . We then vary the latent space size and report the same evaluation metrics. Table 2 shows that models with multiple small latent variables perform significantly better than those with large and few latent variables.
The next question is to interpret the meaning of the learned latent action symbols. To achieve this, the latent action of an utterance is obtained from a greedy mapping: . We set =3 and =5, so that there are at most 125 different latent actions, and each can now be represented by --, e.g. “How are you?” 1-4-2. Assuming that we have access to manually clustered data according to certain classes (e.g. dialog acts), it is unfair to use classic cluster measures Vinh et al. (2010) to evaluate the clusters from latent actions. This is because the uniform prior evenly distribute the data to all possible latent actions, so that it is expected that frequent classes will be assigned to several latent actions. Thus we utilize the homogeneity metric Rosenberg and Hirschberg (2007) that measures if each latent action contains only members of a single class. We tested this on the SW and DD, which contain human annotated features and we report the latent actions’ homogeneity w.r.t these features in Table 3.
On DD, results show DI-VST works better than DI-VAE in terms of creating actions that are more coherent for emotion and dialog acts. The results are interesting on SW since DI-VST performs worse on dialog acts than DI-VAE. One reason is that the dialog acts in SW are more fine-grained (42 acts) than the ones in DD (5 acts) so that distinguishing utterances based on words in is more important than the information in the neighbouring utterances.
We then apply the proposed methods to SMD which has no manual annotation and contains task-oriented dialogs. Two experts are shown 5 randomly selected utterances from each latent action and are asked to give an action name that can describe as many of the utterances as possible. Then an Amazon Mechanical Turk study is conducted to evaluate whether other utterances from the same latent action match these titles. 5 workers see the action name and a different group of 5 utterances from that latent action. They are asked to select all utterances that belong to the given actions, which tests the homogeneity of the utterances falling in the same cluster. Negative samples are included to prevent random selection. Table 4 shows that both methods work well and DI-VST achieved better homogeneity than DI-VAE.
|Model||Exp Agree||Worker||Match Rate|
Since DI-VAE is trained to reconstruct its input and DI-VST is trained to model the context, they group utterances in different ways. For example, DI-VST would group “Can I get a restaurant”, “I am looking for a restaurant” into one action where DI-VAE may denote two actions for them. Finally, Table 5 shows sample annotation results, which show cases of the different types of latent actions discovered by our models.
|DI-VAE||scheduling||- sys: okay, scheduling a yoga activity with Tom for the 8th at 2pm.|
|- sys: okay, scheduling a meeting for 6 pm on Tuesday with your boss to go over the quarterly report.|
|requests||- usr: find out if it ’s supposed to rain|
|- usr: find nearest coffee shop|
|DI-VST||ask schedule info||- usr: when is my football activity and who is going with me?|
|- usr: tell me when my dentist appointment is?|
|requests||- usr: how about other coffee?|
|- usr: 11 am please|
Finally we implement an LAED as follows. The encoder is a hierarchical recurrent encoder Serban et al. (2016) with bi-directional GRU-RNNs as the utterance encoder and a second GRU-RNN as the discourse encoder. The discourse encoder output its last hidden state . The decoder is another GRU-RNN and its initial state of the decoder is obtained by , where comes from the recognition network of the proposed methods. The policy network
is a 2-layer multi-layer perceptron (MLP) that models. We use up to the previous 10 utterances as the dialog context and denote the LAED using DI-VAE latent actions as AE-ED and the one uses DI-VST as ST-ED.
First we need to confirm whether an LAED can generate responses that are consistent with the semantics of a given . To answer this, we use a pre-trained recognition network to check if a generated response carries the attributes in the given action. We generate dialog responses on a test dataset via with greedy RNN decoding. The generated responses are passed into the and we measure attribute accuracy by counting as correct if .
Table 6 shows our generated responses are highly consistent with the given latent actions. Also, latent actions from DI-VAE achieve higher attribute accuracy than the ones from DI-VST, because from auto-encoding is explicitly trained for reconstruction. Adding is effective in forcing the decoder to take into account during its generation, which helps the most in more challenging open-domain chatting data, e.g. SW and DD. The accuracy of ST-ED on SW is worse than the other two datasets. The reason is that SW contains many short utterances that can be either a continuation of the same speaker or a new turn from the other speaker, whereas the responses in the other two domains are always followed by a different speaker. The more complex context pattern in SW may require special treatment. We leave it for future work.
The second experiment checks if the policy network is able to predict the right latent action given just the dialog context. We report both accuracy, i.e. and perplexity of . The perplexity measure is more useful for open domain dialogs because decision-making in complex dialogs is often one-to-many given a similar context Zhao et al. (2017).
|AE-ED||3.045 (51.5% sys 52.4% usr 50.5%)|
|ST-ED||1.695 (75.5% sys 82.1% usr 69.2%)|
|AE-ED||4.47 (35.8%)||4.46 (31.68%)|
|ST-ED||3.89 (47.5%)||3.68 (33.2%)|
Table 7 shows the prediction scores on the three dialog datasets. These scores provide useful insights to understand the complexity of a dialog dataset. For example, accuracy on open-domain chatting is harder than the task-oriented SMD data. Also, it is intuitive that predicting system actions is easier than predicting user actions on SMD. Also, in general the prediction scores for ST-ED are higher the ones for AE-ED. The reason is related to our previous discussion about the granularity of the latent actions. Since latent actions from DI-VST mainly model the the type of utterances used in certain types of context, it is easier for the policy network to predict latent actions from DI-VST. Therefore, choosing the type of latent actions is a design choice and depends on the type of interpretability that is needed. We finish with an example generated from the two variants of LAED on SMD as shown in Table 8
. Given a dialog context, our systems are able to output a probability distribution over different latent actions that have interpretable meaning along with their natural language realizations.
|usr: Where does my friend live?|
|AE-ED||give loc info||- Tom is at 753 University Ave, and a road block.|
|=0.34||- Comfort Inn is at 7 miles away.|
|give user info||- Your home address is 5671 barringer street.|
|=0.22||- Your home is at 10 ames street.|
|ST-ED||give loc info||- Jill’s house is 8 miles away at 347 Alta Mesa Ave.|
|=0.93||- Jill lives at 347 Alta Mesa Ave.|
This paper presents a novel unsupervised framework that enables the discovery of discrete latent actions and interpretable dialog response generation. Our main contributions reside in the two sentence representation models DI-VAE and DI-VST, and their integration with the encoder decoder models. Experiments show the proposed methods outperform strong baselines in learning discrete latent variables and showcase the effectiveness of interpretable dialog response generation. Our findings also suggest promising future research directions, including learning better context-based latent actions and using reinforcement learning to adapt policy networks. We believe that this work is an important step forward towards creating generative dialog models that can not only generalize to large unlabelled datasets in complex domains but also be explainable to human users.
This work was funded by NSF grant CNS- 1512973. The opinions expressed in this paper do not necessarily reflect those of NSF.
Journal of machine Learning research3(Jan):993–1022.
Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL).
Partially observable markov decision processes for spoken dialog systems.Computer Speech & Language 21(2):393–422.
where and is mutual information between and by definition.
Bowman et al., Bowman et al. (2015) have shown that one can transform between two sentences by interpolating in the latent space of continuous VAEs. We found that our DI-VAE enjoys the same property. Specifically, two sentences and are sampled and their latent code are and . We can then interpolate by flipping each latent code from to , . For models with latent variables, one sentence can transform to another one in at most steps. Table 9 shows an example.
|So you can keep record of all the checks you write.|
|So you can get all kinds of information and credit cards.|
|So you can keep track of all the credit cards.|
|So you kind of look at the credit union.|
|So you know of all the credit cards.|
|Yeah because you know of all the credit cards.|
|Right you know at least a lot of times.|
|Right you know a lot of times.|
The details of the three dialog datasets are shown below.
|# of Dialogs||3,031||13,118||2,400|
|Avg Dialog Len||6.36||9.84||59.2|
|Avg Utterance Len||12.1||16.3||22.1|
All RNNs use GRUs Chung et al. (2014). The GRU-RNNs for DI-VAE and DI-VST have hidden size 512. The utterance encoder in LAED has hidden size 256 for one direction and the context encoder and response decoder have hidden size 512. The word embedding is shared everywhere with embedding size 200. The temperature of Gumbel Softmax is set to 1. We train with Adams Kingma and Ba (2014) with initial learning rate 0.001.