Log In Sign Up

Learning Reward Machines: A Study in Partially Observable Reinforcement Learning

by   Rodrigo Toro Icarte, et al.

Reinforcement learning (RL) is a central problem in artificial intelligence. This problem consists of defining artificial agents that can learn optimal behaviour by interacting with an environment – where the optimal behaviour is defined with respect to a reward signal that the agent seeks to maximize. Reward machines (RMs) provide a structured, automata-based representation of a reward function that enables an RL agent to decompose an RL problem into structured subproblems that can be efficiently learned via off-policy learning. Here we show that RMs can be learned from experience, instead of being specified by the user, and that the resulting problem decomposition can be used to effectively solve partially observable RL problems. We pose the task of learning RMs as a discrete optimization problem where the objective is to find an RM that decomposes the problem into a set of subproblems such that the combination of their optimal memoryless policies is an optimal policy for the original problem. We show the effectiveness of this approach on three partially observable domains, where it significantly outperforms A3C, PPO, and ACER, and discuss its advantages, limitations, and broader potential.


page 1

page 2

page 3

page 4


Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning

Reinforcement learning (RL) methods usually treat reward functions as bl...

Policy Gradients Incorporating the Future

Reasoning about the future – understanding how decisions in the present ...

Noisy Symbolic Abstractions for Deep RL: A case study with Reward Machines

Natural and formal languages provide an effective mechanism for humans t...

Defining Admissible Rewards for High Confidence Policy Evaluation

A key impediment to reinforcement learning (RL) in real applications wit...

Induction of Subgoal Automata for Reinforcement Learning

In this work we present ISA, a novel approach for learning and exploitin...

Learning Task Automata for Reinforcement Learning using Hidden Markov Models

Training reinforcement learning (RL) agents using scalar reward signals ...

Learning to Run with Potential-Based Reward Shaping and Demonstrations from Video Data

Learning to produce efficient movement behaviour for humanoid robots fro...