Physically Embedded Planning Problems: New Challenges for Reinforcement Learning

by   Mehdi Mirza, et al.

Recent work in deep reinforcement learning (RL) has produced algorithms capable of mastering challenging games such as Go, chess, or shogi. In these works the RL agent directly observes the natural state of the game and controls that state directly with its actions. However, when humans play such games, they do not just reason about the moves but also interact with their physical environment. They understand the state of the game by looking at the physical board in front of them and modify it by manipulating pieces using touch and fine-grained motor control. Mastering complicated physical systems with abstract goals is a central challenge for artificial intelligence, but it remains out of reach for existing RL algorithms. To encourage progress towards this goal we introduce a set of physically embedded planning problems and make them publicly available. We embed challenging symbolic tasks (Sokoban, tic-tac-toe, and Go) in a physics engine to produce a set of tasks that require perception, reasoning, and motor control over long time horizons. Although existing RL algorithms can tackle the symbolic versions of these tasks, we find that they struggle to master even the simplest of their physically embedded counterparts. As a first step towards characterizing the space of solution to these tasks, we introduce a strong baseline that uses a pre-trained expert game player to provide hints in the abstract space to an RL agent's policy while training it on the full sensorimotor control task. The resulting agent solves many of the tasks, underlining the need for methods that bridge the gap between abstract planning and embodied control.


page 1

page 3

page 4

page 5

page 9


Beating the World's Best at Super Smash Bros. with Deep Reinforcement Learning

There has been a recent explosion in the capabilities of game-playing ar...

Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban

Intelligent robots need to achieve abstract objectives using concrete, s...

Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning

Reinforcement learning (RL) studies how an agent comes to achieve reward...

Learning to Play No-Press Diplomacy with Best Response Policy Iteration

Recent advances in deep reinforcement learning (RL) have led to consider...

Insights From the NeurIPS 2021 NetHack Challenge

In this report, we summarize the takeaways from the first NeurIPS 2021 N...

Option Discovery for Autonomous Generation of Symbolic Knowledge

In this work we present an empirical study where we demonstrate the poss...

PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception

The ability to perceive and reason about social interactions in the cont...

Please sign up or login with your details

Forgot password? Click here to reset