Pavlovian Signalling with General Value Functions in Agent-Agent Temporal Decision Making

01/11/2022
by   Andrew Butcher, et al.
0

In this paper, we contribute a multi-faceted study into Pavlovian signalling – a process by which learned, temporally extended predictions made by one agent inform decision-making by another agent. Signalling is intimately connected to time and timing. In service of generating and receiving signals, humans and other animals are known to represent time, determine time since past events, predict the time until a future stimulus, and both recognize and generate patterns that unfold in time. We investigate how different temporal processes impact coordination and signalling between learning agents by introducing a partially observable decision-making domain we call the Frost Hollow. In this domain, a prediction learning agent and a reinforcement learning agent are coupled into a two-part decision-making system that works to acquire sparse reward while avoiding time-conditional hazards. We evaluate two domain variations: machine agents interacting in a seven-state linear walk, and human-machine interaction in a virtual-reality environment. Our results showcase the speed of learning for Pavlovian signalling, the impact that different temporal representations do (and do not) have on agent-agent coordination, and how temporal aliasing impacts agent-agent and human-agent interactions differently. As a main contribution, we establish Pavlovian signalling as a natural bridge between fixed signalling paradigms and fully adaptive communication learning between two agents. We further show how to computationally build this adaptive signalling process out of a fixed signalling process, characterized by fast continual prediction learning and minimal constraints on the nature of the agent receiving signals. Our results therefore suggest an actionable, constructivist path towards communication learning between reinforcement learning agents.

READ FULL TEXT
research
03/17/2022

The Frost Hollow Experiments: Pavlovian Signalling as a Path to Coordination and Communication Between Agents

Learned communication between agents is a powerful tool when approaching...
research
05/07/2019

Learned human-agent decision-making, communication and joint action in a virtual reality environment

Humans make decisions and act alongside other humans to pursue both shor...
research
11/18/2021

Finding Useful Predictions by Meta-gradient Descent to Improve Decision-making

In computational reinforcement learning, a growing body of work seeks to...
research
06/17/2016

Introspective Agents: Confidence Measures for General Value Functions

Agents of general intelligence deployed in real-world scenarios must ada...
research
02/20/2023

Learning Goal-based Movement via Motivational-based Models in Cognitive Mobile Robots

Humans have needs motivating their behavior according to intensity and c...
research
06/27/2022

Parametrically Retargetable Decision-Makers Tend To Seek Power

If capable AI agents are generally incentivized to seek power in service...

Please sign up or login with your details

Forgot password? Click here to reset