Evaluating Visual Conversational Agents via Cooperative Human-AI Games

by   Prithvijit Chattopadhyay, et al.

As AI continues to advance, human-AI teams are inevitable. However, progress in AI is routinely measured in isolation, without a human in the loop. It is crucial to benchmark progress in AI, not just in isolation, but also in terms of how it translates to helping humans perform certain tasks, i.e., the performance of human-AI teams. In this work, we design a cooperative game - GuessWhich - to measure human-AI team performance in the specific context of the AI being a visual conversational agent. GuessWhich involves live interaction between the human and the AI. The AI, which we call ALICE, is provided an image which is unseen by the human. Following a brief description of the image, the human questions ALICE about this secret image to identify it from a fixed pool of images. We measure performance of the human-ALICE team by the number of guesses it takes the human to correctly identify the secret image after a fixed number of dialog rounds with ALICE. We compare performance of the human-ALICE teams for two versions of ALICE. Our human studies suggest a counterintuitive trend - that while AI literature shows that one version outperforms the other when paired with an AI questioner bot, we find that this improvement in AI-AI performance does not translate to improved human-AI performance. This suggests a mismatch between benchmarking of AI in isolation and in the context of human-AI teams.


page 4

page 7

page 8


Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi

Deep reinforcement learning has generated superhuman AI in competitive g...

Who/What is My Teammate? Team Composition Considerations in Human-AI Teaming

There are many unknowns regarding the characteristics and dynamics of hu...

Conversational AI: The Science Behind the Alexa Prize

Conversational agents are exploding in popularity. However, much work re...

Cognitive Anthropomorphism of AI: How Humans and Computers Classify Images

Modern AI image classifiers have made impressive advances in recent year...

Development of a Trust-Aware User Simulator for Statistical Proactive Dialog Modeling in Human-AI Teams

The concept of a Human-AI team has gained increasing attention in recent...

Towards an AI to Win Ghana's National Science and Maths Quiz

Can an AI win Ghana's National Science and Maths Quiz (NSMQ)? That is th...

Please sign up or login with your details

Forgot password? Click here to reset