Reinforcement Learning with Algorithms from Probabilistic Structure Estimation

by   Jonathan P. Epperlein, et al.

Reinforcement learning (RL) algorithms aim to learn optimal decisions in unknown environments through experience of taking actions and observing the rewards gained. In some cases, the environment is not influenced by the actions of the RL agent, in which case the problem can be modeled as a contextual multi-armed bandit and lightweight myopic algorithms can be employed. On the other hand, when the RL agent's actions affect the environment, the problem must be modeled as a Markov decision process and more complex RL algorithms are required which take the future effects of actions into account. Moreover, in many modern RL settings, it is unknown from the outset whether or not the agent's actions will impact the environment and it is often not possible to determine which RL algorithm is most fitting. In this work, we propose to avoid this dilemma entirely and incorporate a choice mechanism into our RL framework. Rather than assuming a specific problem structure, we use a probabilistic structure estimation procedure based on a likelihood-ratio (LR) test to make a more informed selection of learning algorithm. We derive a sufficient condition under which myopic policies are optimal, present an LR test for this condition, and derive a bound on the regret of our framework. We provide examples of real-world scenarios where our framework is needed and provide extensive simulations to validate our approach.



There are no comments yet.


page 1

page 2

page 3

page 4


Making Sense of Reinforcement Learning and Probabilistic Inference

Reinforcement learning (RL) combines a control problem with statistical ...

5G Handover using Reinforcement Learning

In typical wireless cellular systems, the handover mechanism involves re...

Gap-Dependent Unsupervised Exploration for Reinforcement Learning

For the problem of task-agnostic reinforcement learning (RL), an agent f...

Causal Reinforcement Learning: An Instrumental Variable Approach

In the standard data analysis framework, data is first collected (once f...

Observational Learning by Reinforcement Learning

Observational learning is a type of learning that occurs as a function o...

Reinforcement Learning for Physical Layer Communications

In this chapter, we will give comprehensive examples of applying RL in o...

HTMRL: Biologically Plausible Reinforcement Learning with Hierarchical Temporal Memory

Building Reinforcement Learning (RL) algorithms which are able to adapt ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.