LAVA: Latent Action Spaces via Variational Auto-encoding for Dialogue Policy Optimization

11/18/2020
by   Nurul Lubis, et al.
0

Reinforcement learning (RL) can enable task-oriented dialogue systems to steer the conversation towards successful task completion. In an end-to-end setting, a response can be constructed in a word-level sequential decision making process with the entire system vocabulary as action space. Policies trained in such a fashion do not require expert-defined action spaces, but they have to deal with large action spaces and long trajectories, making RL impractical. Using the latent space of a variational model as action space alleviates this problem. However, current approaches use an uninformed prior for training and optimize the latent distribution solely on the context. It is therefore unclear whether the latent representation truly encodes the characteristics of different actions. In this paper, we explore three ways of leveraging an auxiliary task to shape the latent variable distribution: via pre-training, to obtain an informed prior, and via multitask learning. We choose response auto-encoding as the auxiliary task, as this captures the generative factors of dialogue responses while requiring low computational cost and neither additional data nor labels. Our approach yields a more action-characterized latent representations which support end-to-end dialogue policy optimization and achieves state-of-the-art success rates. These results warrant a more wide-spread use of RL in end-to-end dialogue models.

READ FULL TEXT
research
05/16/2022

Taming Continuous Posteriors for Latent Variational Dialogue Policies

Utilizing amortized variational inference for latent-action reinforcemen...
research
07/07/2021

DORA: Toward Policy Optimization for Task-oriented Dialogue System with Efficient Context

Recently, reinforcement learning (RL) has been applied to task-oriented ...
research
02/23/2019

Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models

Defining action spaces for conversational agents and optimizing their de...
research
04/18/2022

Training and Evaluation of Deep Policies using Reinforcement Learning and Generative Models

We present a data-efficient framework for solving sequential decision-ma...
research
03/10/2021

Causal-aware Safe Policy Improvement for Task-oriented dialogue

The recent success of reinforcement learning's (RL) in solving complex t...
research
09/01/2023

JoTR: A Joint Transformer and Reinforcement Learning Framework for Dialog Policy Learning

Dialogue policy learning (DPL) is a crucial component of dialogue modell...
research
07/24/2022

Anti-Overestimation Dialogue Policy Learning for Task-Completion Dialogue System

A dialogue policy module is an essential part of task-completion dialogu...

Please sign up or login with your details

Forgot password? Click here to reset