DeepAI AI Chat
Log In Sign Up

Evaluating Multimodal Interactive Agents

by   Josh Abramson, et al.

Creating agents that can interact naturally with humans is a common goal in artificial intelligence (AI) research. However, evaluating these interactions is challenging: collecting online human-agent interactions is slow and expensive, yet faster proxy metrics often do not correlate well with interactive evaluation. In this paper, we assess the merits of these existing evaluation metrics and present a novel approach to evaluation called the Standardised Test Suite (STS). The STS uses behavioural scenarios mined from real human interaction data. Agents see replayed scenario context, receive an instruction, and are then given control to complete the interaction offline. These agent continuations are recorded and sent to human annotators to mark as success or failure, and agents are ranked according to the proportion of continuations in which they succeed. The resulting STS is fast, controlled, interpretable, and representative of naturalistic interactions. Altogether, the STS consolidates much of what is desirable across many of our standard evaluation metrics, allowing us to accelerate research progress towards producing agents that can interact naturally with humans.


page 4

page 6

page 18

page 20


Imitating Interactive Intelligence

A common vision from science fiction is that robots will one day inhabit...

Evaluating Empathy in Artificial Agents

The novel research area of computational empathy is in its infancy and m...

Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning

A common vision from science fiction is that robots will one day inhabit...

Proceedings of the AI-HRI Symposium at AAAI-FSS 2018

The goal of the Interactive Learning for Artificial Intelligence (AI) fo...

A modular architecture for creating multimodal agents

The paper describes a flexible and modular platform to create multimodal...

Evaluating Agent Interactions Through Episodic Knowledge Graphs

We present a new method based on episodic Knowledge Graphs (eKGs) for ev...