DeepAI AI Chat
Log In Sign Up

QACE: Asking Questions to Evaluate an Image Caption

by   Hwanhee Lee, et al.
Seoul National University

In this paper, we propose QACE, a new metric based on Question Answering for Caption Evaluation. QACE generates questions on the evaluated caption and checks its content by asking the questions on either the reference caption or the source image. We first develop QACE-Ref that compares the answers of the evaluated caption to its reference, and report competitive results with the state-of-the-art metrics. To go further, we propose QACE-Img, which asks the questions directly on the image, instead of reference. A Visual-QA system is necessary for QACE-Img. Unfortunately, the standard VQA models are framed as a classification among only a few thousand categories. Instead, we propose Visual-T5, an abstractive VQA system. The resulting metric, QACE-Img is multi-modal, reference-less, and explainable. Our experiments show that QACE-Img compares favorably w.r.t. other reference-less metrics. We will release the pre-trained models to compute QACE.


page 2

page 7


An Analysis of Visual Question Answering Algorithms

In visual question answering (VQA), an algorithm must answer text-based ...

Data-QuestEval: A Referenceless Metric for Data to Text Semantic Evaluation

In this paper, we explore how QuestEval, which is a Text-vs-Text metric,...

RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question

Existing metrics for evaluating the quality of automatically generated q...

iVQA: Inverse Visual Question Answering

In recent years, visual question answering (VQA) has become topical as a...

Towards a performance analysis on pre-trained Visual Question Answering models for autonomous driving

This short paper presents a preliminary analysis of three popular Visual...

Evaluation of FEM and MLFEM AI-explainers in Image Classification tasks with reference-based and no-reference metrics

The most popular methods and algorithms for AI are, for the vast majorit...