Learning and Understanding a Disentangled Feature Representation for Hidden Parameters in Reinforcement Learning

11/29/2022
by   Christopher Reale, et al.
0

Hidden parameters are latent variables in reinforcement learning (RL) environments that are constant over the course of a trajectory. Understanding what, if any, hidden parameters affect a particular environment can aid both the development and appropriate usage of RL systems. We present an unsupervised method to map RL trajectories into a feature space where distance represents the relative difference in system behavior due to hidden parameters. Our approach disentangles the effects of hidden parameters by leveraging a recurrent neural network (RNN) world model as used in model-based RL. First, we alter the standard world model training algorithm to isolate the hidden parameter information in the world model memory. Then, we use a metric learning approach to map the RNN memory into a space with a distance metric approximating a bisimulation metric with respect to the hidden parameters. The resulting disentangled feature space can be used to meaningfully relate trajectories to each other and analyze the hidden parameter. We demonstrate our approach on four hidden parameters across three RL environments. Finally we present two methods to help identify and understand the effects of hidden parameters on systems.

READ FULL TEXT

page 1

page 4

page 5

research
11/29/2022

Symmetry Detection in Trajectory Data for More Meaningful Reinforcement Learning Representations

Knowledge of the symmetries of reinforcement learning (RL) systems can b...
research
06/30/2023

Comparing Reinforcement Learning and Human Learning using the Game of Hidden Rules

Reliable real-world deployment of reinforcement learning (RL) methods re...
research
05/14/2021

Estimating Disentangled Belief about Hidden State and Hidden Task for Meta-RL

There is considerable interest in designing meta-reinforcement learning ...
research
11/17/2020

Explaining Conditions for Reinforcement Learning Behaviors from Real and Imagined Data

The deployment of reinforcement learning (RL) in the real world comes wi...
research
10/08/2020

Trajectory Inspection: A Method for Iterative Clinician-Driven Design of Reinforcement Learning Studies

Treatment policies learned via reinforcement learning (RL) from observat...
research
07/07/2020

The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learning

Deep model-based Reinforcement Learning (RL) has the potential to substa...
research
03/22/2018

Learning through deterministic assignment of hidden parameters

Supervised learning frequently boils down to determining hidden and brig...

Please sign up or login with your details

Forgot password? Click here to reset