Measuring Difficulty of Novelty Reaction

07/28/2022
by   Ekaterina Nikonova, et al.
0

Current AI systems are designed to solve close-world problems with the assumption that the underlying world is remaining more or less the same. However, when dealing with real-world problems such assumptions can be invalid as sudden and unexpected changes can occur. To effectively deploy AI-powered systems in the real world, AI systems should be able to deal with open-world novelty quickly. Inevitably, dealing with open-world novelty raises an important question of novelty difficulty. Knowing whether one novelty is harder to deal with than another, can help researchers to train their systems systematically. In addition, it can also serve as a measurement of the performance of novelty robust AI systems. In this paper, we propose to define the novelty reaction difficulty as a relative difficulty of performing the known task after the introduction of the novelty. We propose a universal method that can be applied to approximate the difficulty. We present the approximations of the difficulty using our method and show how it aligns with the results of the evaluation of AI agents designed to deal with novelty.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/16/2021

The Difficulty of Novelty Detection in Open-World Physical Domains: An Application to Angry Birds

Detecting and responding to novel situations in open-world environments ...
research
12/08/2020

A Unifying Framework for Formal Theories of Novelty:Framework, Examples and Discussion

Managing inputs that are novel, unknown, or out-of-distribution is criti...
research
07/04/2019

Procedural Generation of Initial States of Sokoban

Procedural generation of initial states of state-space search problems h...
research
10/14/2022

Hardness of Samples Need to be Quantified for a Reliable Evaluation System: Exploring Potential Opportunities with a New Task

Evaluation of models on benchmarks is unreliable without knowing the deg...
research
11/04/2021

Representation Edit Distance as a Measure of Novelty

Adaptation to novelty is viewed as learning to change and augment existi...
research
03/03/2023

BO-Muse: A human expert and AI teaming framework for accelerated experimental design

In this paper we introduce BO-Muse, a new approach to human-AI teaming f...
research
06/24/2019

Deep Learning in a Computational Model for Conceptual Shifts in a Co-Creative Design System

This paper presents a computational model for conceptual shifts, based o...

Please sign up or login with your details

Forgot password? Click here to reset