DeepAI AI Chat
Log In Sign Up

Importance Weighted Transfer of Samples in Reinforcement Learning

05/28/2018
by   Andrea Tirinzoni, et al.
0

We consider the transfer of experience samples (i.e., tuples < s, a, s', r >) in reinforcement learning (RL), collected from a set of source tasks to improve the learning process in a given target task. Most of the related approaches focus on selecting the most relevant source samples for solving the target task, but then all the transferred samples are used without considering anymore the discrepancies between the task models. In this paper, we propose a model-based technique that automatically estimates the relevance (importance weight) of each source sample for solving the target task. In the proposed approach, all the samples are transferred and used by a batch RL algorithm to solve the target task, but their contribution to the learning process is proportional to their importance weight. By extending the results for importance weighting provided in supervised learning literature, we develop a finite-sample analysis of the proposed batch RL algorithm. Furthermore, we empirically compare the proposed algorithm to state-of-the-art approaches, showing that it achieves better learning performance and is very robust to negative transfer, even when some source tasks are significantly different from the target task.

READ FULL TEXT
02/09/2022

Transferred Q-learning

We consider Q-learning with knowledge transfer, using samples from a tar...
08/31/2011

Transfer from Multiple MDPs

Transfer reinforcement learning (RL) methods leverage on the experience ...
05/26/2020

Time-Variant Variational Transfer for Value Functions

In most transfer learning approaches to reinforcement learning (RL) the ...
03/21/2022

Lean Evolutionary Reinforcement Learning by Multitasking with Importance Sampling

Studies have shown evolution strategies (ES) to be a promising approach ...
07/22/2021

Learning to Transfer: A Foliated Theory

Learning to transfer considers learning solutions to tasks in a such way...
02/26/2020

Generalized Hindsight for Reinforcement Learning

One of the key reasons for the high sample complexity in reinforcement l...
06/14/2023

Provably Efficient Offline Reinforcement Learning with Perturbed Data Sources

Existing theoretical studies on offline reinforcement learning (RL) most...

Code Repositories

iw-transfer-rl

Code for our paper "Importance Weighted Transfer of Samples in Reinforcement Learning" (https://arxiv.org/abs/1805.10886), which was accepted at ICML 2018.


view repo