DeepAI AI Chat
Log In Sign Up

Maximizing the Total Reward via Reward Tweaking

02/09/2020
by   Chen Tessler, et al.
12

In reinforcement learning, the discount factor γ controls the agent's effective planning horizon. Traditionally, this parameter was considered part of the MDP; however, as deep reinforcement learning algorithms tend to become unstable when the effective planning horizon is long, recent works refer to γ as a hyper-parameter. In this work, we focus on the finite-horizon setting and introduce reward tweaking. Reward tweaking learns a surrogate reward function r̃ for the discounted setting, which induces an optimal (undiscounted) return in the original finite-horizon task. Theoretically, we show that there exists a surrogate reward which leads to optimality in the original task and discuss the robustness of our approach. Additionally, we perform experiments in a high-dimensional continuous control task and show that reward tweaking guides the agent towards better long-horizon returns when it plans for short horizons using the tweaked reward.

READ FULL TEXT
12/07/2016

Effect of Reward Function Choices in MDPs with Value-at-Risk

This paper studies Value-at-Risk (VaR) problems in short- and long-horiz...
11/26/2021

Learning Long-Term Reward Redistribution via Randomized Return Decomposition

Many practical applications of reinforcement learning require agents to ...
02/27/2023

Reinforcement Learning with Depreciating Assets

A basic assumption of traditional reinforcement learning is that the val...
09/25/2022

Reward Learning using Structural Motifs in Inverse Reinforcement Learning

The Inverse Reinforcement Learning (IRL) problem has seen rapid evolutio...
01/06/2022

Admissible Policy Teaching through Reward Design

We study reward design strategies for incentivizing a reinforcement lear...
05/31/2022

Hierarchies of Reward Machines

Reward machines (RMs) are a recent formalism for representing the reward...
06/09/2022

Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning

World models in model-based reinforcement learning usually face unrealis...