Neural Self Talk: Image Understanding via Continuous Questioning and Answering

12/10/2015
by   Yezhou Yang, et al.
0

In this paper we consider the problem of continuously discovering image contents by actively asking image based questions and subsequently answering the questions being asked. The key components include a Visual Question Generation (VQG) module and a Visual Question Answering module, in which Recurrent Neural Networks (RNN) and Convolutional Neural Network (CNN) are used. Given a dataset that contains images, questions and their answers, both modules are trained at the same time, with the difference being VQG uses the images as input and the corresponding questions as output, while VQA uses images and questions as input and the corresponding answers as output. We evaluate the self talk process subjectively using Amazon Mechanical Turk, which show effectiveness of the proposed method.

READ FULL TEXT

page 5

page 6

research
03/19/2017

VQABQ: Visual Question Answering by Basic Questions

Taking an image and question as the input of our method, it can output t...
research
11/09/2015

Neural Module Networks

Visual question answering is fundamentally compositional in nature---a q...
research
12/29/2022

Learning One Abstract Bit at a Time Through Self-Invented Experiments Encoded as Neural Networks

There are two important things in science: (A) Finding answers to given ...
research
07/12/2020

Applying recent advances in Visual Question Answering to Record Linkage

Multi-modal Record Linkage is the process of matching multi-modal record...
research
12/15/2021

3D Question Answering

Visual Question Answering (VQA) has witnessed tremendous progress in rec...
research
03/23/2017

Recurrent and Contextual Models for Visual Question Answering

We propose a series of recurrent and contextual neural network models fo...
research
11/18/2015

Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction

We tackle image question answering (ImageQA) problem by learning a convo...

Please sign up or login with your details

Forgot password? Click here to reset