Neuro-Symbolic World Models for Adapting to Open World Novelty

01/16/2023
by   Jonathan Balloch, et al.
0

Open-world novelty–a sudden change in the mechanics or properties of an environment–is a common occurrence in the real world. Novelty adaptation is an agent's ability to improve its policy performance post-novelty. Most reinforcement learning (RL) methods assume that the world is a closed, fixed process. Consequentially, RL policies adapt inefficiently to novelties. To address this, we introduce WorldCloner, an end-to-end trainable neuro-symbolic world model for rapid novelty adaptation. WorldCloner learns an efficient symbolic representation of the pre-novelty environment transitions, and uses this transition model to detect novelty and efficiently adapt to novelty in a single-shot fashion. Additionally, WorldCloner augments the policy learning process using imagination-based adaptation, where the world model simulates transitions of the post-novelty environment to help the policy adapt. By blending ”imagined” transitions with interactions in the post-novelty environment, performance can be recovered with fewer total environment interactions. Using environments designed for studying novelty in sequential decision-making problems, we show that the symbolic world model helps its neural policy adapt more efficiently than model-based and model-based neural-only reinforcement learning methods.

READ FULL TEXT

page 2

page 8

research
03/23/2022

NovGrid: A Flexible Grid World for Evaluating Agent Response to Novelty

A robust body of reinforcement learning techniques have been developed t...
research
06/04/2021

Detecting and Adapting to Novelty in Games

Open-world novelty occurs when the rules of an environment can change ab...
research
11/04/2021

Representation Edit Distance as a Measure of Novelty

Adaptation to novelty is viewed as learning to change and augment existi...
research
12/08/2020

A Unifying Framework for Formal Theories of Novelty:Framework, Examples and Discussion

Managing inputs that are novel, unknown, or out-of-distribution is criti...
research
06/23/2022

NovelCraft: A Dataset for Novelty Detection and Discovery in Open Worlds

In order for artificial agents to perform useful tasks in changing envir...
research
03/24/2023

Learning to Operate in Open Worlds by Adapting Planning Models

Planning agents are ill-equipped to act in novel situations in which the...
research
06/02/2000

Novelty Detection on a Mobile Robot Using Habituation

In this paper a novelty filter is introduced which allows a robot operat...

Please sign up or login with your details

Forgot password? Click here to reset