DeepAI AI Chat
Log In Sign Up

A Taxonomy of Similarity Metrics for Markov Decision Processes

by   Álvaro Visús, et al.

Although the notion of task similarity is potentially interesting in a wide range of areas such as curriculum learning or automated planning, it has mostly been tied to transfer learning. Transfer is based on the idea of reusing the knowledge acquired in the learning of a set of source tasks to a new learning process in a target task, assuming that the target and source tasks are close enough. In recent years, transfer learning has succeeded in making Reinforcement Learning (RL) algorithms more efficient (e.g., by reducing the number of samples needed to achieve the (near-)optimal performance). Transfer in RL is based on the core concept of similarity: whenever the tasks are similar, the transferred knowledge can be reused to solve the target task and significantly improve the learning performance. Therefore, the selection of good metrics to measure these similarities is a critical aspect when building transfer RL algorithms, especially when this knowledge is transferred from simulation to the real world. In the literature, there are many metrics to measure the similarity between MDPs, hence, many definitions of similarity or its complement distance have been considered. In this paper, we propose a categorization of these metrics and analyze the definitions of similarity proposed so far, taking into account such categorization. We also follow this taxonomy to survey the existing literature, as well as suggesting future directions for the construction of new metrics.


page 1

page 2

page 3

page 4


Transferred Q-learning

We consider Q-learning with knowledge transfer, using samples from a tar...

A partition-based similarity for classification distributions

Herein we define a measure of similarity between classification distribu...

Structural Similarity for Improved Transfer in Reinforcement Learning

Transfer learning is an increasingly common approach for developing perf...

Transfer from Multiple MDPs

Transfer reinforcement learning (RL) methods leverage on the experience ...

Metrics and continuity in reinforcement learning

In most practical applications of reinforcement learning, it is untenabl...

Robust Knowledge Transfer in Tiered Reinforcement Learning

In this paper, we study the Tiered Reinforcement Learning setting, a par...

Experience Selection Using Dynamics Similarity for Efficient Multi-Source Transfer Learning Between Robots

In the robotics literature, different knowledge transfer approaches have...