A Corpus-free State2Seq User Simulator for Task-oriented Dialogue

09/10/2019 ∙ by Yutai Hou, et al. ∙ Harbin Institute of Technology Tencent 0

Recent reinforcement learning algorithms for task-oriented dialogue system absorbs a lot of interest. However, an unavoidable obstacle for training such algorithms is that annotated dialogue corpora are often unavailable. One of the popular approaches addressing this is to train a dialogue agent with a user simulator. Traditional user simulators are built upon a set of dialogue rules and therefore lack response diversity. This severely limits the simulated cases for agent training. Later data-driven user models work better in diversity but suffer from data scarcity problem. To remedy this, we design a new corpus-free framework that taking advantage of their benefits. The framework builds a user simulator by first generating diverse dialogue data from templates and then build a new State2Seq user simulator on the data. To enhance the performance, we propose the State2Seq user simulator model to efficiently leverage dialogue state and history. Experiment results on an open dataset show that our user simulator helps agents achieve an improvement of 6.36 State2Seq model outperforms the seq2seq baseline for 1.9 F-score.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 12

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Task-oriented dialogue systems assist users to achieve specific goals such as finding restaurants or booking flights [25]. To learn such a system is very challenging. Recently, reinforcement learning (RL) methods have been introduced due to their advantages in sequential decision. [25, 18, 24, 7]. An RL based dialogue agent can learn from dialogue data or reward signals by interacting with real users. Unfortunately, interacting with real users is costly and there is often no enough data or even no data for new domains. To overcome these obstacles, building user simulators is studied for training RL dialogue algorithms [22, 14].

Figure 1: The proposed corpus-free framework for building user simulators.

User simulators can be divided into two categories: traditional and data-driven user simulator. Traditional user simulators are agenda-based or rule-based [13, 20]. A rule-based user simulator can be built without data, but needs domain-specific knowledge and hard to generalize to new contexts. Besides, the rule-based model lacks response diversity, which largely limits the effectiveness of RL training. Latter data-driven user simulators ease the problem of diversity and depend less on expert knowledge. They imitate user behaviors from datasets with statistical models, such as Bayesian models [17, 8]

, hidden Markov models

[4] and seq2seq models [1]. Statistical user simulators are inherently diverse and often require a large amount of expert-labeled data for training. However, they can not cope with limited data situations.

In this paper, to overcome the data scarcity problem and build a user simulator with sufficient diversity, we propose a new corpus-free framework for building user simulators. It combines the ideas of rule-based and data-driven user simulators. As shown in Figure 1, the framework generates dialogue data from templates and train data-driven user model on it. The template consists of user goals, response strategy and natural language generation rules. An example of the template is shown in Table

1. In addition to templates, data generation uses an RL-based built-in agent to improve data diversity and explore more dialogue cases. The statistical nature of data-driven user simulator provides more diversity than rule-based ones. Diverse responses allow covering more situation for policy training.

To enhance the user simulator’s quality, we propose a novel attention based State2Seq user simulator to leverage the dialogue state and history better. The model first learns representations for dialogue context items. Dialogue context contains structured data of dialogue states, user goals and agent response. Then for each dialogue turn, the model predicts user actions sequentially with the attention on context. Attention helps to pick action more accurately. For example, suppose agent response is (request:movie, inform:date=today), and user goal is (request=[ticket], constraint=[date:tomorrow, movie:Deadpool]). The user model can easily output actions (inform:movie, deny:date) by attending to the agent request and the states of constraint inconsistency respectively.

Experiments are conducted on the movie booking dataset [13] and an in-house restaurant domain dataset. We evaluate both the user simulator model itself and the policy trained by it. On movie booking dataset, our policy achieves an improvement of points on the success rate over the strongest baseline. And proposed State2Seq model outperforms the seq2seq baseline for F-score on response accuracy.

This paper has 3 main contributions: 1. To solve data scarcity, we design a new corpus-free framework for building a user simulator with response diversity. 2. We introduce the attention mechanism to task-oriented user simulator and propose a State2Seq model to get better user behavior modeling. 3. Experiments show that response diversity and attention on dialogue context improve user model and agent policy.

Our code is available at: https://github.com/AtmaHou/UserSimulator

2 Proposed Framework

We focus on developing a user simulator, which is diverse to cover enough dialogue situations. To solve the data scarcity problem, our framework builds a user simulator with only templates and no dialogue data. There are two main components in the framework: a template based data generator and a neural user simulator. The framework (1) first generate data from the templates using a built-in agent, (2) then train a data-driven user simulator on it.

Template Name Template Content
G = =[ movie = Godfather, time = 5 pm ], ,
= [ ticket, theater]
V = if = {req:time} and time in ,
then = inform:time =
N = if = {inform:time=g.C.time} ,
then = ”I want to see it at 5 pm”
Table 1: Template example for movie booking domain. is user goal, is response strategy and is NLG rules. req is request.

2.1 Template Definition

Template is the input to our framework and is used to generate data. We define a template as: , which includes user goals , response strategies and natural language generation rules . An example of movie booking domain template is illustrated in Table1.

is a set of predefined user goals and defined as: , where is the number of goals. Each user goal is defined as which includes a set of user constraints and a set of user requests [20].

is a set of rules for response strategies, which is relatively easy to obtain [13]. It is defined as: , where and are response rules for user and agent respectively. For each , we define it as a function that maps dialogue context to response:

where is the response for the turn. Specifically, as User and agent may take multiple actions in one turn, we define response as a set of single actions: .

is a set of rules for natural language generation(NLG): Each rule maps actions and user goal to natural language and is defined as .

Figure 2: State2Seq user simulator.

2.2 Data Generation

The data generator generates conversation log with templates as input, as shown in Figure 1. There are two steps for the generation: i. Generate basic data with templates only. ii. Generate diverse data with a built-in RL agent and templates.

Generate basic data To start from no corpus, we first collect some rule-based conversation as basic data, which is a common warm-up option [13]. When generating one dialogue, we first pick a user goal from as background and construct a random starting utterance. Then, for each turn, we search the suitable rule to generate the response actions for user and agent. NLG rules is then used to render actions to utterance.

Generate diverse data On the basis of basic data, we generate diverse data. A built-in RL based agent is used here. is warmed up with basic data initially and further trained by interacting with . When generating each dialogue, all process are same to basic data generation, except for agent response actions are given by ’s policy.

We enhance the data diversity by the following operations: i. Leverage the RL exploration mechanism during data generation. ii. Generate data with the built-in agents of different training stages. So both weak and strong policy are used, which allows collecting both clumsy and fluent dialogue.

Feature Description
Constraint Status Status about whether user constraint slots have been informed by user
Request Status Status about whether user request slots have been satisfied by agent
Slot Consistency Status of whether the slot values provided by agent are consistent to user constraints
Dialog Status Dialogue status of success, failed and no outcome yet
Table 2: Feature definition

2.3 User Model Training

After collecting the dialogue data, we train the user simulator on it in supervised style. Given a dialogue context, the user model is trained to predict a set of user actions as response. Only user policy is learned from data here.

3 State2Seq User Simulator

We aim to propose a user simulator that makes better use of dialogue context.

User simulator mimics human responses to the dialogue system output. A user model predicts user response with dialogue context. Given user goal and dialogue history , it predict turn response as :

where . Following [1], we formulate the action selection as a sequence generation problem:

However, dialogue history can be very long. So it is very hard for the user model to leverage history information directly. To remedy this, we extract the key information as dialogue state from the dialogue history. We then define the dialogue context as a combination of dialogue state and user goals.

For better usage of dialogue context, we learn vector representations for each context items. And attention mechanism is proposed to leverage context more clearly.

Figure 2 shows the structure of the State2Seq user simulator. The State2Seq model maintains a dialogue state for tracking the dialogue history. It uses an encoding module to provide vector representations for dialogue context items. For each turn, the model integrates those representations into a context representation H. Then the model decodes H into a sequence of actions as output. Attention helps the model use dialogue context during decoding.

3.1 Dialogue Context Representation

The main idea of dialogue context representation is to refine history information. Forgetting useless information is proven to be important for data-driven model.

The dialogue context consists of dialogue state and user goal. Dialogue state includes last turn agent response and state features. Following [1], we extract the state features explained in Table 2. Constraint Status, Request Status, and Dialog Status are used to help the user simulator track the progress of the current conversation, and Slot Consistency allows the user simulator to correct the agent on wrong information. Each feature is recorded as a status vector , where is the number of slot types and could be 1, 0, -1 for active, irrelevant and negative.

Attention mechanism relies on a good representation of items. Representation is also important for sequence decoding, as its need a good initial state . So we propose an encoding module to learn vector representations for context items. To help to learn of representation, we share representations for the common slots in context items. Some negative status is rare in the corpus, which makes it hard to learn good representations. To remedy this, we only learn vector representations for positive slot status, and then use the corresponding inverse vector to represent negative status.

H is obtained by dimension reduction of context representation, which can further forget irrelevant information. Formally, given a agent response , the user goal and current dialogue history , State2Seq updates dialogue state and represent context with vectors. is then obtained as:

where denotes vectors concatenation and means fetching vector representations for items in .

3.2 Action Generation with Attention

We formulate actions selection as sequence generation process with attention. Sequence generation provides diversity in selecting actions. Attention mechanism helps to use dialogue context information directly. Specifically, we generate responses by attending on items in user goal , dialogue state and the last agent response .

During actions decoding, for each time step

, LSTM provides hidden representation

, where denotes the LSTM hidden state at time step , the is cell state and is input.

The attention weight of item in dialogue context is calculated as: where is semantic representation vector of item. Attention is used to calculate the decoding output , where is the representations of all items in context.

Then we model the distribution the time step ’s action :

And the user action is predicted as:

4 Experiment

4.1 Dataset

We used two datasets in our experiments: movie ticket booking data and restaurant reservation data. The movie ticket booking data is an open task-completion dialogue dataset proposed by [13]. For each dialogue, the system gathers information about the customer’s desires and books the movie tickets. The success or failure of dialogue is assessed based on (1) whether a movie is booked, and (2) whether the movie satisfies the user’s constraints. The data includes 11 dialogue acts, 29 slots, 277 user goal templates. Rule templates for response strategy and NLG are also included in [13]’s work.

To test the method’s generation ability, we also build a dataset for restaurant reservation domain. In each dialogue, the user reserves a table under his/her requirements. The data includes 11 dialogue acts, 24 slots and 184 user goal templates. We design rule templates for response strategy and NLG based on the ones in [13]’s work.

4.2 Evaluation

We evaluate a user simulator by: i. evaluation of the user simulator itself. ii. evaluation of the policy trained with it.

Evaluation of Agent Policy The main value of simulator is to train agent policy. We use both human and automatic evaluation for policy here. A DQN model is used as the agent to learn the policy.

We adopt cross-model evaluation proposed by [19]. user models are first used to train policies. Each policy is then tested against different user simulators. Finally, we calculate the average of scores. A policy trained by a good simulator can still perform well on poor simulators [19, 11]. A higher average score indicates a better simulator ability for training agent. For metric, we use success rate, average reward and average turn number, which have been widely accepted as a standard metric of multi-turn agent [6, 2, 13].

Evaluation of User Simulator We evaluate the user simulator itself from two aspects.

Firstly, following [1], we evaluate the accuracy of predicting actions. F-score is used as the metric.

Secondly, a user simulator’s generalization ability is also important. It has more tolerance for exploration of training agent. We compare different user simulators’ such ability by making conversation against a same rule agent. User model with better ability should achieve more success rate.

4.3 Model Details

For the State2Seq model, we set embedding size as 300. We use a 2 layer LSTMs for decoding with hidden state size of 256. During the training, we set the batch size as 32, a dropout as 0.8 and teacher forcing rate of 0.5. The learning rate is set as 0.001 and we set a learning rate decay of 0.9.

For the RL agent model used in data generation and evaluation, we use the DQN model. We set experience pool size as 1000, hidden layer size as 80. Experience replay redesigned for dialogue setting is applied. We use

-greedy exploration of 0.01. The learning batch size is 16. We use 100 warm-up epochs and 500 training epochs. Model simulates 100 dialogues for each epoch.

We also simply extend our model by replacing the sequence decoder with a multi-label classifier. We name it as State2MLC. State2MLC takes dialogue state, user goal, agent response as input and predict multiple actions. State2MLC is trained with Multi-Label Soft Margin Loss.

4.4 Baselines

We compared with the following baselines:

  • Seq2Seq is user simulator proposed by [1]. It extracts history turns’ features as input sequence and decodes action sequence.

  • Seq2Seq-att is based on Seq2Seq model and adds attention mechanism over input sequence.

  • Agenda based user simulator is proposed by [20]. It is corpus free and generates user response by maintaining a user agenda with rule. We use the agenda user simulator provided by [13].

Model Movie Booking Restaurant Reservation
Avg. Succ. Avg. Rwd. Avg. Turns Avg. Succ. Avg. Rwd. Avg. Turns
State2MLC 0.487 8.55 21.82 0.305 -17.22 29.69
State2Seq 0.551 14.17 25.85 0.524 11.77 24.21
Seq2Seq 0.412 -3.49 27.77 0.501 6.23 29.83
Seq2Seq-Att 0.430 -2.59 30.39 0.514 9.67 26.20
Agenda 0.438 -2.88 32.88 0.508 10.88 22.17
Table 3: Evaluation of agent policy trained by different user model. Results above dash-line are from our model, which achieve best performance in most task.
Model Avg. Succ. Avg. Rwd. Avg. Turns
State2Seq (Ours) 0.778 53.88 11.88
Agenda 0.571 22.29 14.57
Table 4: Human evaluation of trained agent
Model Succ. Avg. Rwd. Avg. Turns
State2MLC (Ours) 0.628 26.19 16.36
State2Seq (Ours) 0.800 48.39 17.23
Seq2Seq 0.462 2.98 28.92
Seq2Seq-Att 0.480 2.11 32.98
Agenda 0.814 50.36 15.66
Table 5: Analysis of agent policy performance during training on movie domain. At beginning of each policy training epoch, we test the policy’s performance. The results reflect the models’ overfitting to training set.
Model Action Accuracy Generalization Ability Test
Movie Restaurant Movie Restaurant
F1 F1 Avg. Succ. Avg. Rwd. Avg. Succ. Avg. Rwd.
State2MLC (Ours) 0.704 0.695 0.442 6.06 0.436 5.34
State2Seq (Ours) 0.711 0.683 0.400 5.51 0.484 11.09
Seq2Seq 0.692 0.662 0.063 -31.87 0.126 -31.87
Seq2Seq-Att 0.705 0.677 0.000 -46.99 0.000 -46.99
Agenda N/A N/A 0.392 0.04 0.410 2.20
Table 6: Evaluation of user simulator model.

4.5 Performance of Agent Policy

We compare the policy trained by different user simulators with cross-model evaluation.

Results on movie booking data Table 3 shows the evaluation results on movie booking domain. The results show that the policy obtained by our model outperforms baselines.

On average success rate, policy trained by our State2Seq model outperforms the Agenda model for points. As the Agenda [13] model is rule-based, the main difference between State2Seq and it is that State2Seq has more diverse responses. This demonstrates that user simulator diversity improves policy ability for finishing task and generalization. Policy trained by our model outperforms the Agenda model on success rate and average reward.

The results show that the policy trained by other statistical methods underperform the Agenda based model. Because those user simulators lack for response accuracy, which will mislead and confuse the policy training. Comparing Seq2Seq-att to Seq2Seq, the results show the effectiveness of attention mechanism. And State2Seq’s improvements over the Seq2Seq-att show that the refined context representation of the State2Seq model does help the response generation.

Results on restaurant data Table 3 shows the cross evaluation results for restaurant reservation domain. The results show our method could work consistently well in different domains. Most models score higher on data in the restaurant domain, because the field is relatively simpler and has fewer slots. The State2MLC model does not perform well. This is due to the fact that State2MLC model has a much simpler structure, so it is likely to overfit to generated data in a simple domain and limits the generalization ability of the policy.

Human evaluation We perform a human evaluation on the movie domain, and each agent is tested by chatting with 2 domain experts for 50 dialogues. Table 4 shows that the agents trained in our model can be better adapted to the real situation.

Analysis on policy training process Table 5 shows the evaluation of training process. We perform testing at each training epoch, and report the averaged score. The policy is evaluated against the environment for training. Policy trained by our model outperforms the ones from statistical user simulator, which reflects that our user simulator improves the training performance of agent policy. It is because our user simulator has better generalization to respond to agent’s unreasonable actions in the early training stage. On the other hand, diversity helps RL algorithm training. Agenda achieve good scores as rule environment is relatively easy for overfitting.

4.6 Performance of User Simulator

User model’s performance is mainly reflected by the ability of predicting user responses.

Action accuracy. Table 6 shows the models’ accuracy of predicting user actions. Our model achieves the best performance, outperforming the seq2seq model [1] for points on F-score. The results also show a correlation between the evaluation of action prediction and the agent policy’s performance, which demonstrates that user simulator quality affects agents performance.

The improvements mainly come from two aspects: Firstly, the attention mechanism provides specific context information. Secondly, refined context representation filters the useless information. The Seq2Seq-att model outperforms the Seq2Seq model by adding the attention mechanism. This reflects the effectiveness of the attention mechanism. By comparing State2Seq to Seq2Seq-att, improvement shows that due to forgetting mechanism, refined context representation is more effective than sequence encoder.

Analysis of generalization ability The generalization ability is also important for a user simulator in agent training. We compare different user simulators’ generalization by making them chat with a same rule-based agent. Table 6 shows the results and State2Seq and MLC2Seq are optimal in all user simulators. Our best model outperforms the Agenda for and points success rate on the 2 domains. As the user simulators are trained to mimic user rules in the template, the improvement reflects that our framework can generate new diverse dialogues data to avoid user simulator overfitting to user response strategy in the template. Other user simulators perform worse on this test, we address this to the fact that these methods are less accurate in generating user actions and rule-based agent has a low allowance for response error.

Agenda User Simulator
usr: Which theater is available? act:req, req slots: {theater}
agt: Which theater would you like? act:req, req slots: {theater}
usr: Which theater is available? act:req, req slots: {theater}
agt: Which theater would you like? act:req, req slots: {theater}
(loop…)
Our User Simulator
usr: Which theater is available? act:req, req slots:{theater}
agt: Which theater would you like? act:req, req slots:{theater}
usr: I want to watch at Seattle. act:inform, inform slots:{city: Seattle}
agt: Seattle is available. act:inform, inform: {city: Seattle}
usr: Which theater is available? act:req, req slots: {theater}
agt: The Pearl Theater is available. act:inform, inform slots: {theater: The Pearl Theater}
Table 7: Case study of the difference between our user simulator and agenda user simulator. Here, user is requesting for ticket and theater, and user’s constraints are { movie name: deadpool, city: Seattle , num: 2, date: tomorrow }

Context: Actions:
agt:How many tickets do you need? act:req, inform S:{}, req S:{# of people}
usr:I want 2 tickets please! act:inform, inform S:{# of people:2}, req S:{}
Figure 3: Visualization of attention. The vertical axis is the generated user response and the horizontal axis is the dialogue context. The lighter color means higher attention. S denotes slots.

4.7 Case Study and Visualization

To find out the difference between rule-based model and the proposed model. We perform case study on dialogues between the user simulator and an agent. The comparison is shown in Table 7. When agents and users are unable to satisfy each other, the fixed rules of Agenda user simulator may be trapped in the loop shown in the example. But the response from our user simulator is diverse and uncertain. It can try other actions to jump out of the endless loop. The difference shows that our proposed framework for building user simulators has successfully improved the response diversity.

To demonstrate the effectiveness of the attention mechanism over context, we provide a visualization example shown in Figure 3. The figure shows that the attention mechanism successfully learns the correlation between the generated actions and the context. Specifically, when generating the dialogue action of inform and the inform slots, the proposed model pays a higher attention to context items of sys_request_slots and user_goal.

5 Related Work

There is little work solving the data insufficient problem of user simulator. [11] proposed a user simulator that generates user utterance directly, which could ease the effort of user semantic annotation. However, their work is corpus-based and still needs a large corpus to train on it. Other data scarcity problems for task-oriented dialogue system are also investigated. [26] leverage the idea of zero shot learning [16]. They solve the problem of dialogue generation for a new domain by mapping actions to latent space. [12] and [9] provided data augmentation methods for language understanding. [23] proposed to build new domain agent efficiently by machine self-play and crowdsourcing.

The first user simulator was proposed by [5]. There are two kinds of user simulators in terms of working levels. User simulators of semantic level interact to agent with dialogue acts and corresponding slot-value pairs. [21, 3, 1, 6]. User simulators of utterance level communicates to agent with utterance directly [10, 11, 15]. Our user simulator can work on both of the two levels.

6 Conclusion

In this paper, we study the problem of building user simulators for task-oriented dialogue from templates with no corpus. We solve the data scarcity and increase simulator response diversity by proposing a corpus-free framework. In our framework, we generate diverse data with only templates and trains a data-driven user simulator on it. To predict user response more accurately, we proposed a novel State2Seq user model. It predicts user response with attention on refined dialogue context representations. Experiment results show that with more response diversity, our user simulator improves the agent policy by 6.36% success rate. Attention and refined context representation help the State2Seq model outperform Seq2Seq baseline for 1.9 F-score.

Acknowledgments

We are grateful for helpful comments and suggestions from the anonymous reviewers. This work was supported by the National Natural Science Foundation of China (NSFC) via grant 61632011, 61772153 and 61772156.

References

  • [1] L. E. Asri, J. He, and K. Suleman (2016) A sequence-to-sequence model for user simulation in spoken dialogue systems. See DBLP:conf/interspeech/2016, pp. 1151–1155. External Links: Link, Document Cited by: §1, §3.1, §3, 1st item, §4.2, §4.6, §5.
  • [2] I. Casanueva, P. Budzianowski, P. Su, N. Mrksic, T. Wen, S. Ultes, L. M. Rojas-Barahona, S. J. Young, and M. Gasic (2017) A benchmarking environment for reinforcement learning based task oriented dialogue management. CoRR abs/1711.11023. External Links: Link, 1711.11023 Cited by: §4.2.
  • [3] S. Chandramohan, M. Geist, F. Lefevre, and O. Pietquin (2011) User simulation in dialogue systems using inverse reinforcement learning. In Proc. of Interspeech, Cited by: §5.
  • [4] H. Cuayáhuitl, S. Renals, O. Lemon, and H. Shimodaira (2005) Human-computer dialogue simulation using hidden markov models. In Proc. of ASRU, pp. 290–295. Cited by: §1.
  • [5] W. Eckert, E. Levin, and R. Pieraccini (1997) User modeling for spoken dialogue system evaluation. In Proc. of ASRU, Vol. 97. Cited by: §5.
  • [6] J. Gao, K. Wong, B. Peng, J. Liu, and X. Li (2018) Deep dyna-q: integrating planning for task-completion dialogue policy learning. See DBLP:conf/acl/2018-1, pp. 2182–2192. External Links: Link Cited by: §4.2, §5.
  • [7] M. Gašić and S. Young (2014) Gaussian processes for pomdp-based dialogue manager optimization. IEEE/ACM Trans. on Audio, Speech & Language Processing 22 (1), pp. 28–40. Cited by: §1.
  • [8] K. Georgila, J. Henderson, and O. Lemon (2005) Learning user simulations for information state update dialogue systems. In Proc. of Eurospeech, Cited by: §1.
  • [9] Y. Hou, Y. Liu, W. Che, and T. Liu (2018) Sequence-to-sequence data augmentation for dialogue language understanding. Proc. of COLING. Cited by: §5.
  • [10] S. Jung, C. Lee, K. Kim, M. Jeong, and G. G. Lee (2009) Data-driven user simulation for automated evaluation of spoken dialog systems. Computer Speech & Language 23 (4), pp. 479–509. Cited by: §5.
  • [11] F. Kreyssig, I. Casanueva, P. Budzianowski, and M. Gasic (2018) Neural user simulation for corpus-based policy optimisation of spoken dialogue systems. See DBLP:conf/sigdial/2018, pp. 60–69. External Links: Link Cited by: §4.2, §5, §5.
  • [12] G. Kurata, B. Xiang, and B. Zhou (2016) Labeled data generation with encoder-decoder LSTM for semantic slot filling. See DBLP:conf/interspeech/2016, pp. 725–729. External Links: Link, Document Cited by: §5.
  • [13] X. Li, Y. Chen, L. Li, and J. Gao (2017) End-to-End Task-Completion Neural Dialogue Systems. In Proc. of IJCNLP, Note: 微软那篇增强学习代码的论文 External Links: 1703.01008, Link Cited by: §1, §1, §2.1, §2.2, 3rd item, §4.1, §4.1, §4.2, §4.5.
  • [14] X. Li, Z. C. Lipton, B. Dhingra, L. Li, J. Gao, and Y. Chen (2016) A user simulator for task-completion dialogues. arXiv preprint arXiv:1612.05688. Cited by: §1.
  • [15] B. Liu and I. Lane (2017) Iterative policy learning in end-to-end trainable task-oriented neural dialog models. In ASRU Workshop, pp. 482–489. Cited by: §5.
  • [16] M. Palatucci, D. Pomerleau, G. E. Hinton, and T. M. Mitchell (2009) Zero-shot learning with semantic output codes. In Proc. of NIPS, Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta (Eds.), pp. 1410–1418. External Links: Link Cited by: §5.
  • [17] O. Pietquin and T. Dutoit (2006) A probabilistic framework for dialog simulation and optimal strategy learning. IEEE Trans. on Audio, Speech, and Language Processing 14 (2), pp. 589–599. Cited by: §1.
  • [18] N. Roy, J. Pineau, and S. Thrun (2000) Spoken dialogue management using probabilistic reasoning. See DBLP:conf/acl/2000, External Links: Link Cited by: §1.
  • [19] J. Schatzmann, M. N. Stuttle, W. Karl, and S. Young (2005) Effects of the user model on simulation-based learning of dialogue strategies. In Proc. of ASRU, Cited by: §4.2.
  • [20] J. Schatzmann, B. Thomson, K. Weilhammer, H. Ye, and S. Young (2007) Agenda-based user simulation for bootstrapping a pomdp dialogue system. In Proc. of NAACL, pp. 149–152. Cited by: §1, §2.1, 3rd item.
  • [21] J. Schatzmann, B. Thomson, and S. Young (2007) Statistical user simulation with a hidden agenda. Proc. of SIGDial 273282 (9). Cited by: §5.
  • [22] J. Schatzmann, K. Weilhammer, M. Stuttle, and S. Young (2006) A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies.

    The knowledge engineering review

    21 (2), pp. 97–126.
    Cited by: §1.
  • [23] P. Shah, D. Hakkani-Tür, G. Tür, A. Rastogi, A. Bapna, N. Nayak, and L. Heck (2018)

    Building a conversational agent overnight with dialogue self-play

    .
    arXiv preprint arXiv:1801.04871. Cited by: §5.
  • [24] J. D. Williams and S. Young (2007)

    Partially observable markov decision processes for spoken dialog systems

    .
    Computer Speech & Language 21 (2), pp. 393–422. Cited by: §1.
  • [25] S. Young, M. Gašić, B. Thomson, and J. D. Williams (2013) Pomdp-based statistical spoken dialog systems: a review. Proc. of the IEEE 101 (5), pp. 1160–1179. Cited by: §1.
  • [26] T. Zhao and M. Eskénazi (2018) Zero-shot dialog generation with cross-domain latent actions. CoRR abs/1805.04803. External Links: Link, 1805.04803 Cited by: §5.