On Value Functions and the Agent-Environment Boundary

05/30/2019

∙

When function approximation is deployed in reinforcement learning (RL), the same problem may be formulated in different ways, often by treating a pre-processing step as a part of the environment or as part of the agent. As a consequence, fundamental concepts in RL, such as (optimal) value functions, are not uniquely defined as they depend on where we draw this agent-environment boundary, causing problems in theoretical analyses that provide optimality guarantees. We address this issue via a simple and novel boundary-invariant analysis of Fitted Q-Iteration, a representative RL algorithm, where the assumptions and the guarantees are invariant to the choice of boundary. We also discuss closely related issues on state resetting and Monte-Carlo Tree Search, deterministic vs stochastic systems, imitation learning, and the verifiability of theoretical assumptions from data.

READ FULL TEXT

On Value Functions and the Agent-Environment Boundary

Offline Reinforcement Learning Under Value and Density-Ratio Realizability: the Power of Gaps

All You Need Is Supervised Learning: From Imitation Learning to Meta-RL With Upside Down RL

Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning

Nonparametric General Reinforcement Learning

Dynamic penalty function approach for constraints handling in reinforcement learning

Scale Invariant Solutions for Overdetermined Linear Systems with Applications to Reinforcement Learning

Interactive Learning from Activity Description

On Value Functions and the Agent-Environment Boundary

Related Research

Offline Reinforcement Learning Under Value and Density-Ratio Realizability: the Power of Gaps

All You Need Is Supervised Learning: From Imitation Learning to Meta-RL With Upside Down RL

Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning

Nonparametric General Reinforcement Learning

Dynamic penalty function approach for constraints handling in reinforcement learning

Scale Invariant Solutions for Overdetermined Linear Systems with Applications to Reinforcement Learning

Interactive Learning from Activity Description