The relationship between dynamic programming and active inference: the discrete, finite-horizon case

09/17/2020
by   Lancelot Da Costa, et al.
18

Active inference is a normative framework for generating behaviour based upon the free energy principle, a theory of self-organisation. This framework has been successfully used to solve reinforcement learning and stochastic control problems, yet, the formal relation between active inference and reward maximisation has not been fully explicated. In this paper, we consider the relation between active inference and dynamic programming under the Bellman equation, which underlies many approaches to reinforcement learning and control. We show that, on partially observable Markov decision processes, dynamic programming is a limiting case of active inference. In active inference, agents select actions to minimise expected free energy. In the absence of ambiguity about states, this reduces to matching expected states with a target distribution encoding the agent's preferences. When target states correspond to rewarding states, this maximises expected reward, as in reinforcement learning. When states are ambiguous, active inference agents will choose actions that simultaneously minimise ambiguity. This allows active inference agents to supplement their reward maximising (or exploitative) behaviour with novelty-seeking (or exploratory) behaviour. This clarifies the connection between active inference and reinforcement learning, and how both frameworks may benefit from each other.

READ FULL TEXT
research
06/07/2020

Sophisticated Inference

Active inference offers a first principle account of sentient behaviour,...
research
09/21/2021

Active inference, Bayesian optimal design, and expected utility

Active inference, a corollary of the free energy principle, is a formal ...
research
07/18/2020

Modulation of viability signals for self-regulatory control

We revisit the role of instrumental value as a driver of adaptive behavi...
research
07/02/2023

On efficient computation in active inference

Despite being recognized as neurobiologically plausible, active inferenc...
research
06/27/2023

An analytical model of active inference in the Iterated Prisoner's Dilemma

This paper addresses a mathematically tractable model of the Prisoner's ...
research
08/22/2022

Deriving time-averaged active inference from control principles

Active inference offers a principled account of behavior as minimizing a...

Please sign up or login with your details

Forgot password? Click here to reset