Learning Agent State Online with Recurrent Generate-and-Test

12/30/2021
by   Amir Samani, et al.
0

Learning continually and online from a continuous stream of data is challenging, especially for a reinforcement learning agent with sequential data. When the environment only provides observations giving partial information about the state of the environment, the agent must learn the agent state based on the data stream of experience. We refer to the state learned directly from the data stream of experience as the agent state. Recurrent neural networks can learn the agent state, but the training methods are computationally expensive and sensitive to the hyper-parameters, making them unideal for online learning. This work introduces methods based on the generate-and-test approach to learn the agent state. A generate-and-test algorithm searches for state features by generating features and testing their usefulness. In this process, features useful for the agent's performance on the task are preserved, and the least useful features get replaced with newly generated features. We study the effectiveness of our methods on two online multi-step prediction problems. The first problem, trace conditioning, focuses on the agent's ability to remember a cue for a prediction multiple steps into the future. In the second problem, trace patterning, the agent needs to learn patterns in the observation signals and remember them for future predictions. We show that our proposed methods can effectively learn the agent state online and produce accurate predictions.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/13/2022

What Should I Know? Using Meta-gradient Descent for Predictive Feature Discovery in a Single Stream of Experience

In computational reinforcement learning, a growing body of work seeks to...
research
11/09/2020

Testbeds for Reinforcement Learning

We present three problems modeled after animal learning experiments desi...
research
11/06/2020

Learning Online Data Association

When an agent interacts with a complex environment, it receives a stream...
research
11/15/2022

Agent-State Construction with Auxiliary Inputs

In many, if not every realistic sequential decision-making task, the dec...
research
04/07/2020

Online Constrained Model-based Reinforcement Learning

Applying reinforcement learning to robotic systems poses a number of cha...
research
11/18/2021

Finding Useful Predictions by Meta-gradient Descent to Improve Decision-making

In computational reinforcement learning, a growing body of work seeks to...
research
06/27/2019

Emergence of Exploratory Look-Around Behaviors through Active Observation Completion

Standard computer vision systems assume access to intelligently captured...

Please sign up or login with your details

Forgot password? Click here to reset