Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

09/29/2015
by   Shakir Mohamed, et al.
0

The mutual information is a core statistical quantity that has applications in all areas of machine learning, whether this is in training of density models over multiple data modalities, in maximising the efficiency of noisy transmission channels, or when learning behaviour policies for exploration by artificial agents. Most learning algorithms that involve optimisation of the mutual information rely on the Blahut-Arimoto algorithm --- an enumerative algorithm with exponential complexity that is not suitable for modern machine learning applications. This paper provides a new approach for scalable optimisation of the mutual information by merging techniques from variational inference and deep learning. We develop our approach by focusing on the problem of intrinsically-motivated learning, where the mutual information forms the definition of a well-known internal drive known as empowerment. Using a variational lower bound on the mutual information, combined with convolutional networks for handling visual input streams, we develop a stochastic optimisation algorithm that allows for scalable information maximisation and empowerment-based reasoning directly from pixels to actions.

READ FULL TEXT
06/04/2020

A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learning

In this paper, we propose a maximum mutual information (MMI) framework f...
10/11/2018

Empowerment-driven Exploration using Mutual Information Estimation

Exploration is a difficult challenge in reinforcement learning and is of...
11/06/2019

Conditional Mutual Information Neural Estimator

Several recent works in communication systems have proposed to leverage ...
06/10/2020

On the Maximum Mutual Information Capacity of Neural Architectures

We derive the closed-form expression of the maximum mutual information -...
05/08/2019

Data-Efficient Mutual Information Neural Estimator

Measuring Mutual Information (MI) between high-dimensional, continuous, ...
12/31/2019

On the Difference Between the Information Bottleneck and the Deep Information Bottleneck

Combining the Information Bottleneck model with deep learning by replaci...
04/26/2020

Notes on Icebreaker

Icebreaker [1] is new research from MSR that is able to achieve state of...