NovelCraft: A Dataset for Novelty Detection and Discovery in Open Worlds

06/23/2022
by   Patrick Feeney, et al.
0

In order for artificial agents to perform useful tasks in changing environments, they must be able to both detect and adapt to novelty. However, visual novelty detection research often only evaluates on repurposed datasets such as CIFAR-10 originally intended for object classification. This practice restricts novelties to well-framed images of distinct object types. We suggest that new benchmarks are needed to represent the challenges of navigating an open world. Our new NovelCraft dataset contains multi-modal episodic data of the images and symbolic world-states seen by an agent completing a pogo-stick assembly task within a video game world. In some episodes, we insert novel objects that can impact gameplay. Novelty can vary in size, position, and occlusion within complex scenes. We benchmark state-of-the-art novelty detection and generalized category discovery models with a focus on comprehensive evaluation. Results suggest an opportunity for future research: models aware of task-specific costs of different types of mistakes could more effectively detect and adapt to novelty in open worlds.

READ FULL TEXT
research
06/04/2021

Detecting and Adapting to Novelty in Games

Open-world novelty occurs when the rules of an environment can change ab...
research
02/28/2023

Methods and Mechanisms for Interactive Novelty Handling in Adversarial Environments

Learning to detect, characterize and accommodate novelties is a challeng...
research
01/16/2023

Neuro-Symbolic World Models for Adapting to Open World Novelty

Open-world novelty–a sudden change in the mechanics or properties of an ...
research
12/23/2022

Human Activity Recognition in an Open World

Managing novelty in perception-based human activity recognition (HAR) is...
research
12/08/2020

A Unifying Framework for Formal Theories of Novelty:Framework, Examples and Discussion

Managing inputs that are novel, unknown, or out-of-distribution is criti...
research
05/13/2021

Handwriting Recognition with Novelty

This paper introduces an agent-centric approach to handle novelty in the...
research
09/21/2023

Environment-biased Feature Ranking for Novelty Detection Robustness

We tackle the problem of robust novelty detection, where we aim to detec...

Please sign up or login with your details

Forgot password? Click here to reset