A Continuous Information Gain Measure to Find the Most Discriminatory Problems for AI Benchmarking

09/09/2018
by   Matthew Stephenson, et al.
0

This paper introduces an information-theoretic method for selecting a small subset of problems which gives us the most information about a group of problem-solving algorithms. This method was tested on the games in the General Video Game AI (GVGAI) framework, allowing us to identify a smaller set of games that still gives a large amount of information about the game-playing agents. This approach can be used to make agent testing more efficient in the future. We can achieve almost as good discriminatory accuracy when testing on only a handful of games as when testing on more than a hundred games, something which is often computationally infeasible. Furthermore, this method can be extended to study the dimensions of effective variance in game design between these games, allowing us to identify which games differentiate between agents in the most complementary ways. As a side effect of this investigation, we provide an up-to-date comparison on agent performance for all GVGAI games, and an analysis of correlations between scores and win-rates across both games and agents.

READ FULL TEXT

page 3

page 4

research
01/31/2018

Deceptive Games

Deceptive games are games where the reward structure or other aspects of...
research
06/04/2018

Shallow decision-making analysis in General Video Game Playing

The General Video Game AI competitions have been the testing ground for ...
research
10/05/2022

Atari-5: Distilling the Arcade Learning Environment down to Five Games

The Arcade Learning Environment (ALE) has become an essential benchmark ...
research
08/18/2023

Preference-conditioned Pixel-based AI Agent For Game Testing

The game industry is challenged to cope with increasing growth in demand...
research
02/12/2019

NAIL: A General Interactive Fiction Agent

Interactive Fiction (IF) games are complex textual decision making probl...
research
11/29/2017

Happiness Pursuit: Personality Learning in a Society of Agents

Modeling personality is a challenging problem with applications spanning...
research
12/27/2022

Teamwork under extreme uncertainty: AI for Pokemon ranks 33rd in the world

The highest grossing media franchise of all times, with over $90 billion...

Please sign up or login with your details

Forgot password? Click here to reset