Constrained Policy Optimization for Controlled Self-Learning in Conversational AI Systems

09/17/2022
by   Mohammad Kachuee, et al.
0

Recently, self-learning methods based on user satisfaction metrics and contextual bandits have shown promising results to enable consistent improvements in conversational AI systems. However, directly targeting such metrics by off-policy bandit learning objectives often increases the risk of making abrupt policy changes that break the current user experience. In this study, we introduce a scalable framework for supporting fine-grained exploration targets for individual domains via user-defined constraints. For example, we may want to ensure fewer policy deviations in business-critical domains such as shopping, while allocating more exploration budget to domains such as music. Furthermore, we present a novel meta-gradient learning approach that is scalable and practical to address this problem. The proposed method adjusts constraint violation penalty terms adaptively through a meta objective that encourages balanced constraint satisfaction across domains. We conduct extensive experiments using data from a real-world conversational AI on a set of realistic constraint benchmarks. Based on the experimental results, we demonstrate that the proposed approach is capable of achieving the best balance between the policy value and constraint satisfaction rate.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/17/2023

Scalable and Safe Remediation of Defective Actions in Self-Learning Conversational Systems

Off-Policy reinforcement learning has been a driving force for the state...
research
04/14/2022

Scalable and Robust Self-Learning for Skill Routing in Large-Scale Conversational AI Systems

Skill routing is an important component in large-scale conversational sy...
research
10/21/2020

Self-Supervised Contrastive Learning for Efficient User Satisfaction Prediction in Conversational Agents

Turn-level user satisfaction is one of the most important performance me...
research
02/19/2021

Probabilistically Guaranteed Satisfaction of Temporal Logic Constraints During Reinforcement Learning

We present a novel reinforcement learning algorithm for finding optimal ...
research
05/24/2022

Meta Policy Learning for Cold-Start Conversational Recommendation

Conversational recommender systems (CRS) explicitly solicit users' prefe...
research
03/05/2021

Automatic Exploration Process Adjustment for Safe Reinforcement Learning with Joint Chance Constraint Satisfaction

In reinforcement learning (RL) algorithms, exploratory control inputs ar...
research
09/27/2022

Follow The Rules: Online Signal Temporal Logic Tree Search for Guided Imitation Learning in Stochastic Domains

Seamlessly integrating rules in Learning-from-Demonstrations (LfD) polic...

Please sign up or login with your details

Forgot password? Click here to reset