Action and Perception as Divergence Minimization

09/03/2020
by   Danijar Hafner, et al.
10

We introduce a unified objective for action and perception of intelligent agents. Extending representation learning and control, we minimize the joint divergence between the combined system of agent and environment and a target distribution. Intuitively, such agents use perception to align their beliefs with the world, and use actions to align the world with their beliefs. Minimizing the joint divergence to an expressive target maximizes the mutual information between the agent's representations and inputs, thus inferring representations that are informative of past inputs and exploring future inputs that are informative of the representations. This lets us explain intrinsic objectives, such as representation learning, information gain, empowerment, and skill discovery from minimal assumptions. Moreover, interpreting the target distribution as a latent variable model suggests powerful world models as a path toward highly adaptive agents that seek large niches in their environments, rendering task rewards optional. The framework provides a common language for comparing a wide range of objectives, advances the understanding of latent variables for decision making, and offers a recipe for designing novel objectives. We recommend deriving future agent objectives the joint divergence to facilitate comparison, to point out the agent's target distribution, and to identify the intrinsic objective terms needed to reach that distribution.

READ FULL TEXT
research
03/11/2021

Understanding the origin of information-seeking exploration in probabilistic objectives for control

The exploration-exploitation trade-off is central to the description of ...
research
12/01/2020

Mutual Information Constraints for Monte-Carlo Objectives

A common failure mode of density models trained as variational autoencod...
research
06/19/2019

Unsupervised State Representation Learning in Atari

State representation learning, or the ability to capture latent generati...
research
10/28/2021

Wasserstein Distance Maximizing Intrinsic Control

This paper deals with the problem of learning a skill-conditioned policy...
research
06/21/2018

Expanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop

Active inference is an ambitious theory that treats perception, inferenc...
research
12/21/2020

Evaluating Agents without Rewards

Reinforcement learning has enabled agents to solve challenging tasks in ...
research
04/13/2018

PAMPC: Perception-Aware Model Predictive Control for Quadrotors

We present a perception-aware model predictive control framework for qua...

Please sign up or login with your details

Forgot password? Click here to reset