Strong Asymptotic Optimality in General Environments

03/04/2019
by   Michael K. Cohen, et al.
8

Reinforcement Learning agents are expected to eventually perform well. Typically, this takes the form of a guarantee about the asymptotic behavior of an algorithm given some assumptions about the environment. We present an algorithm for a policy whose value approaches the optimal value with probability 1 in all computable probabilistic environments, provided the agent has a bounded horizon. This is known as strong asymptotic optimality, and it was previously unknown whether it was possible for a policy to be strongly asymptotically optimal in the class of all computable probabilistic environments. Our agent, Inquisitive Reinforcement Learner (Inq), is more likely to explore the more it expects an exploratory action to reduce its uncertainty about which environment it is in, hence the term inquisitive. Exploring inquisitively is a strategy that can be applied generally; for more manageable environment classes, inquisitiveness is tractable. We conducted experiments in "grid-worlds" to compare the Inquisitive Reinforcement Learner to other weakly asymptotically optimal agents.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/05/2020

Curiosity Killed the Cat and the Asymptotically Optimal Agent

Reinforcement learners are agents that learn to pick actions that lead t...
research
07/27/2011

Asymptotically Optimal Agents

Artificial general intelligence aims to create agents capable of learnin...
research
09/16/2016

A Formal Solution to the Grain of Truth Problem

A Bayesian agent acting in a multi-agent environment learns to predict t...
research
09/29/2012

Optimistic Agents are Asymptotically Optimal

We use optimism to introduce generic asymptotically optimal reinforcemen...
research
02/25/2016

Thompson Sampling is Asymptotically Optimal in General Environments

We discuss a variant of Thompson sampling for nonparametric reinforcemen...
research
06/15/2020

Pessimism About Unknown Unknowns Inspires Conservatism

If we could define the set of all bad outcomes, we could hard-code an ag...
research
06/30/2018

Modeling Friends and Foes

How can one detect friendly and adversarial behavior from raw data? Dete...

Please sign up or login with your details

Forgot password? Click here to reset