1 Introduction
In a Spoken Dialogue System (SDS), one of the main problems is to find appropriate system behaviour for any given situation. This problem is often modelled using reinforcement learning (RL) where the task is to find an optimal policy which maps the current belief state
—an estimate of the user goal— to the next system action
. To do this, RL algorithms seek to optimize an objective function, the reward , using sample dialogues. In contrast to other RL tasks (like AlphaGo Silver et al. (2016)), the reward used in goaloriented dialogue systems usually consists of more than one objective (e.g., task success and dialogue length Levin et al. (1998); Lemon et al. (2006); Young et al. (2013)).However, balancing these rewards is rarely considered and the goal of this paper is to propose a structured method for finding the optimal weights for a multiple objective reward function. Finding a good balance between multiple objectives is usually domainspecific and not straightforward. For example, in the case of task success and dialogue length, if the reward for success is too high, the learning algorithm is insensitive to potentially irritating actions such as repeat provided that the dialogue is ultimately successful. Conversely, if the reward for success is too small, the resulting policy may irritate users by offering inappropriate solutions before fully illiciting the user’s requirements.
In this paper, we propose to find a suitable reward balance by searching through the space of reward component weights. Doing this with conventional RL techniques is infeasible as a policy must be trained for each candidate balance and this requires an enormous number of training dialogues. To alleviate this, we propose to use multiobjective RL (MORL) which is specifically designed for this task (among others Roijers et al. (2013)). Then, only one policy needs to be trained which may be evaluated with several candidate balances. To the best of our knowledge, this is the first time MORL has been applied to dialogue policy optimization.
In contrast to previous work which explicitly selects component weights to maximize user satisfaction Walker (2000) explicitly, the proposed method enables optimisation of an implicit goal by allowing the interplay each reward component to be explored at low computational cost.
Several different algorithms have previously been used for MORL Castelletti et al. (2013); Van Moffaert et al. (2015); Pirotta et al. (2015); Mossalam et al. (2016). In this work, we propose a novel MORL algorithm based on Gaussian processes. This is described in Section 2 along with a brief introduction to MORL. In Section 3, the proposed method for finding a good reward balance with MORL is presented. Section 4 describes the application and evaluation of the balancing method on six different domains. Finally conclusions are drawn in Section 5.
2 Multiobjective Reinforcement Learning with Gaussian Processes
In this Section we present our proposed extension of the GPSARSA algorithm for MORL after giving a brief introduction to single and multiobjective RL and the GPSARSA algorithm itself.
Reinforcement Learning
Reinforcement learning (RL) is used in a sequential decisionmaking process where a decisionmodel (the policy ) is trained based on sample data and a potentially delayed objective signal (the reward ) Sutton and Barto (1998). Implementing the Markov assumption, the policy selects the next action based on the current system belief state to optimise the accumulated future reward at time :
(1) 
Here, denotes the number of future steps, a discount factor and the reward at time .
The function models the expected accumulated future reward when taking action in belief state and then following policy :
(2) 
Gpsarsa
For most realworld problems, finding the exact optimal values is not feasible. Instead, Engel et al. (2005) have proposed the GPSARSA algorithm which uses Gaussian processes (GP) to approximate the function. Gašić and Young (2014) have shown that this works well when applied to the problem of spoken dialogue policy optimisation. GPSARSA is a Bayesian online learning algorithm which models the function as a zeromean GP which is fully defined by a mean and a kernel function :
(3) 
where the kernel models the correlation between data points. Based on sample data, the GP is trained to approximate
such that the variance derived from the kernel represents the uncertainty of the approximation.
In dialogue management, the following kernel has been successfully used:
(4) 
It consists of a linear kernel for the continuous belief representation and the kernel for the discrete system action .
Multiobjective Reinforcement Learning
In multiobjective reinforcement learning (MORL), the objective function does not consist of only one but of many dimensions. Thus, the reward
becomes a vector
, where is the number of objectives.To define the contribution of each objective, a scalarization function is introduced which uses weights for the different objectives to map the vector representation to a scalar value. The solution to a MORL problem is a set of optimal policies containing an optimal policy for any given weight configuration.
In MORL, the function may either be modelled as a vector of functions or directly as the expectation of the scalarized vector of :
(5) 
In practice, the scalarization function is often modelled as a linear function (the weighted sum):
(6) 
Multiobjective GPSARSA
The proposed multiobjective (MO) GPSARSA is based on Equation 5. By approximating the scalarized function directly using a GP, the GPSARSA algorithm may be applied for MORL. The GP (and thus the function) is extended by one parameter—the weight vector : .
Approximating the function with a GP relies on the fact that the accumulated future reward (Eq. 1) may be decomposed as
(7) 
Accordingly, for using a GP to directly estimate the scalarized reward in MOGPSARSA, the equation
(8) 
must hold. This is true in case of using a linear scalarization function (Eq. 6).
To alter the kernel accordingly, a linear kernel for is added to the state kernel^{1}^{1}1A similar type of kernel extension has been proposed previously in a different context, e.g., Casanueva et al. (2015). resulting in
(9) 
Since a linear scalarization function is applied, the correlations with other data points are also assumed to be linear.
To train a policy using multiobjective GPSARSA, a new weight configuration is sampled randomly for each training dialogue. An example of the training process being applied to dialogue policy optimization with the two objectives task success and dialogue length is depicted in Algorithm LABEL:alg:mogpsarsa.
algocf[t]
3 Reward Balancing using MORL
The main contribution of this paper is to provide a structured method for finding a good balance between multiple rewards for learning dialogue policies. For the twoobjective problem of having a task success reward and a dialogue length reward , , the scalarized reward is
(10) 
where is the number of turns and iff the dialogue is successful, zero otherwise.
To find a good reward balance, we adopt the following procedure:

Set initial reward values and along with the initial weight configuration.

Apply MORL to train a policy for a given number of training dialogues and evaluate with different weight configurations.

Select an appropriate balance based on successweight and lengthweight curves to optimise the individual implicit goal.
The method may be refined by applying it recursively with different grid sizes. After selecting a suitable weight configuration, a singleobjective policy may be trained.
4 Experiments and Results
The reward balancing method described in the previous section is applied to six domains: finding TVs, laptops, restaurants or hotels (the latter two in Cambridge and San Francisco). The following table depicts the domain statistics with the number of search constraints, the number of informational items the user can request, and the number of database entities:
Domain  # constr.  # requests  # entities 

CamRestaurants  3  9  110 
CamHotels  5  11  33 
SFRestaurants  6  11  271 
SFHotels  6  10  182 
TV  6  14  94 
Laptops  11  21  126 
For consistency with previous work Gašić and Young (2014); Young et al. (2013); Su et al. (2016) the rewards and are used representing the weight configuration . This results in and .
For the evaluation, simulated dialogues were created using the statistical spoken dialogue toolkit PyDial Ultes et al. (2017). It contains an agendabased user simulator Schatzmann and Young (2009) with an error model to simulate the semantic error rate (SER) encountered in real systems due to the noisy speech channel.
A policy has been trained for each domain using multiobjective GPSARSA with 3,000 dialogues and an SER of 15%. Each policy was evaluated with 300 dialogues for each weight configuration in . The results in Figure 1 are the averages of five trained policies with different random seeds. All curves follow a similar pattern: at some point, the success curve reaches a plateau where the performance does not increase any further with higher .
The following weights were selected: CamRestaurants ; CamHotels ; SFRestaurants ; SFHotels ; TV ; Laptops . These weights were selected by hand according to the success rate^{2}^{2}2Taking into account the overall performance and the proximity to the edge of the plateau. To compensate for possible inaccuracies of the MOGPSARSA, the configuration right at the edge has not been chosen. as well as the average dialogue length.
The selected weights were scaled to keep the turn penalty constant at . Using these reward settings, each domain was evaluated with 4,000 dialogues in 10 batches. After each batch, the policies were evaluated with 300 dialogues. The final results shown in Table 1 (selection of learning curves in Figure 2) are compared to the baseline of (i.e. standard unoptimised reward component weight balance). Evidently, optimising the balance has a significant impact on the performance of the trained polices.
TSR  # Turns  

base.  opt.  base.  opt.  
CamRestaurants  14  88.8%  86.2%  6.4  6.3 
CamHotels  30  75.1%  79.8%  8.1  8.2 
SFRestaurants  47  62.4%  65.7%  8.5  9.1 
SFHotels  30  66.7%  69.4%  8.0  8.0 
TV  30  75.7%  80.5%  7.4  7.4 
Laptops  47  44.6%  54.6%  7.5  8.7 
To analyse the performance of multiobjective GPSARSA, policies were trained and evaluated for each reward balance with singleobjective (SO) GPSARSA (see Figure 1) after the weights had been selected. Each SO policy was trained with 1,000 dialogues and evaluated with 300 dialogues, all averaged over five runs. The successweight curves for SORL clearly resemble the MORL curves for almost all domains except for CamRestaurants where it leads to an incorrect selection of weights. This may be attributed to the kernel used for multiobjective GPSARSA.
It is worth noting that for the presented full MORL analysis, 3,000 training dialogues were necessary for each domain to find a good balance. This is significantly less than the 9,000 dialogues needed for the SORL analysis and this difference would increase further for a finer grain search grid.
5 Conclusion
In this work, we have addressed the problem of finding a good balance between multiple rewards for learning dialogue policies. We have shown the relevance of the problem and demonstrated the usefulness of multiobjective reinforcement learning to facilitate the search for a suitable balance. Using the proposed procedure, only one policy needs to be trained which can then be evaluated for an arbitrary number of reward balances thus drastically reducing the total amount of training dialogues needed.
We have proposed and employed an extension of the GPSARSA algorithm for multiple objectives and applied it to six domains. The experiments show the successful application of our method: the optimal balance improved task success without unduly impacting on dialogue length in all domains except CamRestaurants, where it is clear that the weight selection criteria failed. In practice, this could have been easily trapped by applying a minimum weight to the success criteria. Furthermore, the domaindependence of the reward balance has been confirmed.
For future work, the accuracy of the proposed multiobjective GPSARSA will be further improved with the ultimate goal of using the proposed method to directly learn a multiobjective policy through interaction with real users. To achieve this, alternative weight kernels will be explored. The resulting multiobjective policy may then directly be applied (without the need of retraining a singleobjective policy) and the weights may even be adjusted according to a specific situation or user preferences.
Future work will also include an automatic method to find the optimal balance as well as investigating the relationship between the optimal success reward value and the domain characteristics (similar to Papangelis et al. (2017)).
Acknowledgments
TsungHsien Wen, Paweł Budzianowski and Stefan Ultes are supported by Toshiba Research Europe Ltd, Cambridge Research Laboratory. This research was partly funded by the EPSRC grant EP/M018946/1 Open Domain Statistical Spoken Dialogue Systems.
Data
All experiments were run in simulation. The corresponding source code is included in the PyDial toolkit which can be found on www.pydial.org.
References
 Casanueva et al. (2015) Inigo Casanueva, Thomas Hain, Heidi Christensen, Ricard Marxer, and Phil Green. 2015. Knowledge transfer between speakers for personalised dialogue management. In 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue. page 12.
 Castelletti et al. (2013) A Castelletti, F Pianesi, and M Restelli. 2013. A multiobjective reinforcement learning approach to water resources systems operation: Pareto frontier approximation in a single run. Water Resources Research 49(6):3476–3486.

Engel et al. (2005)
Yaakov Engel, Shie Mannor, and Ron Meir. 2005.
Reinforcement learning with gaussian processes.
In
Proceedings of the 22nd international conference on Machine learning
. ACM, pages 201–208.  Gašić and Young (2014) Milica Gašić and Steve J. Young. 2014. Gaussian processes for POMDPbased dialogue manager optimization. IEEE/ACM Transactions on Audio, Speech, and Language Processing 22(1):28–40.
 Lemon et al. (2006) Oliver Lemon, Kallirroi Georgila, James Henderson, and Matthew Stuttle. 2006. An isu dialogue system exhibiting reinforcement learning of dialogue policies: generic slotfilling in the talk incar system. In Proceedings of the Eleventh Conference of the European Chapter of the Association for Computational Linguistics: Posters & Demonstrations. Association for Computational Linguistics, pages 119–122.

Levin et al. (1998)
Esther Levin, Roberto Pieraccini, and Wieland Eckert. 1998.
Using markov decision process for learning dialogue strategies.
In Acoustics, Speech and Signal Processing, 1998. Proceedings of the 1998 IEEE International Conference on. IEEE, volume 1, pages 201–204.  Mossalam et al. (2016) Hossam Mossalam, Yannis M. Assael, Diederik M Roijers, and Shimon Whiteson. 2016. Multiobjective deep reinforcement learning. CoRR abs/1610.02707. http://arxiv.org/abs/1610.02707.
 Papangelis et al. (2017) Alexandros Papangelis, Stefan Ultes, and Yannis Stylianou. 2017. Domain complexity and policy learning in taskoriented dialogue systems. In Proceedings of the 8th International Workshop On Spoken Dialogue Systems (IWSDS).

Pirotta et al. (2015)
Matteo Pirotta, Simone Parisi, and Marcello Restelli. 2015.
Multiobjective reinforcement learning with continuous pareto
frontier approximation.
In
Proceedings of the TwentyNinth AAAI Conference on Artificial Intelligence
. pages 2928–2934.  Roijers et al. (2013) Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. 2013. A survey of multiobjective sequential decisionmaking. Journal of Artificial Intelligence Research (JAIR) 48:67–113.
 Schatzmann and Young (2009) Jost Schatzmann and Steve J. Young. 2009. The hidden agenda user simulation model. Audio, Speech, and Language Processing, IEEE Transactions on 17(4):733–747.

Silver et al. (2016)
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George
Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda
Panneershelvam, Marc Lanctot, et al. 2016.
Mastering the game of go with deep neural networks and tree search.
Nature 529(7587):484–489.  Su et al. (2016) PeiHao Su, M. Gašić, N. Mrkšić, L. RojasBarahona, Stefan Ultes, D. Vandyke, T. H. Wen, and S. Young. 2016. Online active reward learning for policy optimisation in spoken dialogue systems. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pages 2431–2441.
 Sutton and Barto (1998) Richard S. Sutton and Andrew G. Barto. 1998. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, USA, 1st edition. http://portal.acm.org/citation.cfm?id=551283.
 Ultes et al. (2017) Stefan Ultes, Lina M. RojasBarahona, PeiHao Su, David Vandyke, Dongho Kim, Iñigo Casanueva, Paweł Budzianowski, Nikola Mrkšić, TsungHsien Wen, Milica Gašić, and Steve J. Young. 2017. Pydial: A multidomain statistical dialogue system toolkit. In ACL Demo. Association of Computational Linguistics.
 Van Moffaert et al. (2015) Kristof Van Moffaert, Tim Brys, and Ann Nowé. 2015. Risksensitivity through multiobjective reinforcement learning. In Evolutionary Computation (CEC), 2015 IEEE Congress on. IEEE, pages 1746–1753.
 Walker (2000) Marilyn Walker. 2000. An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. Journal of Artificial Intelligence Research 12:387–416.
 Young et al. (2013) Steve J. Young, Milica Gašić, Blaise Thomson, and Jason D. Williams. 2013. POMDPbased statistical spoken dialog systems: A review. Proceedings of the IEEE 101(5):1160–1179.