Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review

05/02/2018
by   Sergey Levine, et al.
0

The framework of reinforcement learning or optimal control provides a mathematical formalization of intelligent decision making that is powerful and broadly applicable. While the general form of the reinforcement learning problem enables effective reasoning about uncertainty, the connection between reinforcement learning and inference in probabilistic models is not immediately obvious. However, such a connection has considerable value when it comes to algorithm design: formalizing a problem as probabilistic inference in principle allows us to bring to bear a wide array of approximate inference tools, extend the model in flexible and powerful ways, and reason about compositionality and partial observability. In this article, we will discuss how a generalization of the reinforcement learning or optimal control problem, which is sometimes termed maximum entropy reinforcement learning, is equivalent to exact probabilistic inference in the case of deterministic dynamics, and variational inference in the case of stochastic dynamics. We will present a detailed derivation of this framework, overview prior work that has drawn on this and related ideas to propose new reinforcement learning and control algorithms, and describe perspectives on future research.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/06/2022

Optimal Control as Variational Inference

In this article we address the stochastic and risk sensitive optimal con...
research
07/13/2023

Safe Reinforcement Learning as Wasserstein Variational Inference: Formal Methods for Interpretability

Reinforcement Learning or optimal control can provide effective reasonin...
research
09/20/2010

Approximate Inference and Stochastic Optimal Control

We propose a novel reformulation of the stochastic optimal control probl...
research
12/07/2019

From Reinforcement Learning to Optimal Control: A unified framework for sequential decisions

There are over 15 distinct communities that work in the general area of ...
research
08/25/2019

Tutorial and Survey on Probabilistic Graphical Model and Variational Inference in Deep Reinforcement Learning

Probabilistic Graphical Modeling and Variational Inference play an impor...
research
06/23/2020

On the Relationship Between Active Inference and Control as Inference

Active Inference (AIF) is an emerging framework in the brain sciences wh...
research
03/27/2013

ABC Reinforcement Learning

This paper introduces a simple, general framework for likelihood-free Ba...

Please sign up or login with your details

Forgot password? Click here to reset