Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow

03/26/2021
by   John McLeod, et al.
0

In the past decade, model-free reinforcement learning (RL) has provided solutions to challenging domains such as robotics. Model-based RL shows the prospect of being more sample-efficient than model-free methods in terms of agent-environment interactions, because the model enables to extrapolate to unseen situations. In the more recent past, model-based methods have shown superior results compared to model-free methods in some challenging domains with non-linear state transitions. At the same time, it has become apparent that RL is not market-ready yet and that many real-world applications are going to require model-based approaches, because model-free methods are too sample-inefficient and show poor performance in early stages of training. The latter is particularly important in industry, e.g. in production systems that directly impact a company's revenue. This demonstrates the necessity for a toolbox to push the boundaries for model-based RL. While there is a plethora of toolboxes for model-free RL, model-based RL has received little attention in terms of toolbox development. Bellman aims to fill this gap and introduces the first thoroughly designed and tested model-based RL toolbox using state-of-the-art software engineering practices. Our modular approach enables to combine a wide range of environment models with generic model-based agent classes that recover state-of-the-art algorithms. We also provide an experiment harness to compare both model-free and model-based agents in a systematic fashion w.r.t. user-defined evaluation metrics (e.g. cumulative reward). This paves the way for new research directions, e.g. investigating uncertainty-aware environment models that are not necessarily neural-network-based, or developing algorithms to solve industrially-motivated benchmarks that share characteristics with real-world problems.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/25/2018

Temporal Difference Models: Model-Free Deep RL for Model-Based Control

Model-free reinforcement learning (RL) is a powerful, general tool for l...
research
04/19/2021

Adaptive learning for financial markets mixing model-based and model-free RL for volatility targeting

Model-Free Reinforcement Learning has achieved meaningful results in sta...
research
10/14/2022

Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm

During initial iterations of training in most Reinforcement Learning (RL...
research
05/03/2022

RLFlow: Optimising Neural Network Subgraph Transformation with World Models

We explored the use of reinforcement learning (RL) agents that can learn...
research
07/23/2020

Reinforcement Learning Assisted Load Test Generation for E-Commerce Applications

Background: End-user satisfaction is not only dependent on the correct f...
research
03/14/2021

Progressive residual learning for single image dehazing

The recent physical model-free dehazing methods have achieved state-of-t...
research
01/11/2019

An investigation of model-free planning

The field of reinforcement learning (RL) is facing increasingly challeng...

Please sign up or login with your details

Forgot password? Click here to reset