Marginalized Operators for Off-policy Reinforcement Learning

03/30/2022
by   Yunhao Tang, et al.
0

In this work, we propose marginalized operators, a new class of off-policy evaluation operators for reinforcement learning. Marginalized operators strictly generalize generic multi-step operators, such as Retrace, as special cases. Marginalized operators also suggest a form of sample-based estimates with potential variance reduction, compared to sample-based estimates of the original multi-step operators. We show that the estimates for marginalized operators can be computed in a scalable way, which also generalizes prior results on marginalized importance sampling as special cases. Finally, we empirically demonstrate that marginalized operators provide performance gains to off-policy evaluation and downstream policy optimization algorithms.

READ FULL TEXT

page 8

page 24

research
06/27/2023

Value-aware Importance Weighting for Off-policy Reinforcement Learning

Importance sampling is a central idea underlying off-policy prediction i...
research
06/24/2021

Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation

Model-agnostic meta-reinforcement learning requires estimating the Hessi...
research
03/13/2020

Taylor Expansion Policy Optimization

In this work, we investigate the application of Taylor expansions in rei...
research
05/21/2018

A General Family of Robust Stochastic Operators for Reinforcement Learning

We consider a new family of operators for reinforcement learning with th...
research
10/02/2020

Self-Play Reinforcement Learning for Fast Image Retargeting

In this study, we address image retargeting, which is a task that adjust...
research
10/02/2022

GFlowNets and variational inference

This paper builds bridges between two families of probabilistic algorith...
research
11/15/2022

Automatic Evaluation of Excavator Operators using Learned Reward Functions

Training novice users to operate an excavator for learning different ski...

Please sign up or login with your details

Forgot password? Click here to reset