DeepAI AI Chat
Log In Sign Up

Playing Games in the Dark: An approach for cross-modality transfer in reinforcement learning

by   Rui Silva, et al.

In this work we explore the use of latent representations obtained from multiple input sensory modalities (such as images or sounds) in allowing an agent to learn and exploit policies over different subsets of input modalities. We propose a three-stage architecture that allows a reinforcement learning agent trained over a given sensory modality, to execute its task on a different sensory modality-for example, learning a visual policy over image inputs, and then execute such policy when only sound inputs are available. We show that the generalized policies achieve better out-of-the-box performance when compared to different baselines. Moreover, we show this holds in different OpenAI gym and video game environments, even when using different multimodal generative models and reinforcement learning algorithms.


page 4

page 6


How to Sense the World: Leveraging Hierarchy in Multimodal Perception for Robust Reinforcement Learning Agents

This work addresses the problem of sensing the world: how to learn a mul...

Sensory Modality Mapping for Game Adaptation and Design

In this paper we examine methods for taking game-related information pro...

Agents that Listen: High-Throughput Reinforcement Learning with Multiple Sensory Systems

Humans and other intelligent animals evolved highly sophisticated percep...

Multi-modal Feedback for Affordance-driven Interactive Reinforcement Learning

Interactive reinforcement learning (IRL) extends traditional reinforceme...

Agent-State Construction with Auxiliary Inputs

In many, if not every realistic sequential decision-making task, the dec...

Impact Makes a Sound and Sound Makes an Impact: Sound Guides Representations and Explorations

Sound is one of the most informative and abundant modalities in the real...

Learning to Navigate in Complex Environments

Learning to navigate in complex environments with dynamic elements is an...