Benchmark Visual Question Answer Models by using Focus Map

01/13/2018
by   Wenda Qiu, et al.
0

Inferring and Executing Programs for Visual Reasoning proposes a model for visual reasoning that consists of a program generator and an execution engine to avoid end-to-end models. To show that the model actually learns which objects to focus on to answer the questions, the authors give a visualization of the norm of the gradient of the sum of the predicted answer scores with respect to the final feature map. However, the authors do not evaluate the efficiency of focus map. This paper purposed a method for evaluating it. We generate several kinds of questions to test different keywords. We infer focus maps from the model by asking these questions and evaluate them by comparing with the segmentation graph. Furthermore, this method can be applied to any model if focus maps can be inferred from it. By evaluating focus map of different models on the CLEVR dataset, we will show that CLEVR-iep model has learned where to focus more than end-to-end models.

READ FULL TEXT

page 1

page 2

page 3

page 5

research
05/10/2017

Inferring and Executing Programs for Visual Reasoning

Existing methods for visual reasoning attempt to directly map inputs to ...
research
03/26/2022

Learning to Answer Questions in Dynamic Audio-Visual Scenarios

In this paper, we focus on the Audio-Visual Question Answering (AVQA) ta...
research
10/20/2020

SOrT-ing VQA Models : Contrastive Gradient Learning for Improved Consistency

Recent research in Visual Question Answering (VQA) has revealed state-of...
research
03/19/2022

ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning

Charts are very popular for analyzing data. When exploring charts, peopl...
research
09/03/2019

Data Interpretation over Plots

Reasoning over plots by question answering (QA) is a challenging machine...
research
04/16/2022

What If: Generating Code to Answer Simulation Questions

Many texts, especially in chemistry and biology, describe complex proces...

Please sign up or login with your details

Forgot password? Click here to reset