Dynamic Language Binding in Relational Visual Reasoning

04/30/2020
by   Thao Minh Le, et al.
6

We present Language-binding Object Graph Network, the first neural reasoning method with dynamic relational structures across both visual and textual domains with applications in visual question answering. Relaxing the common assumption made by current models that the object predicates pre-exist and stay static, passive to the reasoning process, we propose that these dynamic predicates expand across the domain borders to include pair-wise visual-linguistic object binding. In our method, these contextualized object links are actively found within each recurrent reasoning step without relying on external predicative priors. These dynamic structures reflect the conditional dual-domain object dependency given the evolving context of the reasoning through co-attention. Such discovered dynamic graphs facilitate multi-step knowledge combination and refinements that iteratively deduce the compact representation of the final answer. The effectiveness of this model is demonstrated on image question answering demonstrating favorable performance on major VQA datasets. Our method outperforms other methods in sophisticated question-answering tasks wherein multiple object relations are involved. The graph structure effectively assists the progress of training, and therefore the network learns efficiently compared to other reasoning models.

READ FULL TEXT

page 2

page 3

page 8

research
03/06/2022

Dynamic Key-value Memory Enhanced Multi-step Graph Reasoning for Knowledge-based Visual Question Answering

Knowledge-based visual question answering (VQA) is a vision-language tas...
research
07/10/2019

Learning to Reason with Relational Video Representation for Question Answering

How does machine learn to reason about the content of a video in answeri...
research
03/30/2018

DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer

We present a novel Dynamic Differentiable Reasoning (DDR) framework for ...
research
02/25/2020

Hierarchical Conditional Relation Networks for Video Question Answering

Video question answering (VideoQA) is challenging as it requires modelin...
research
05/10/2019

Language-Conditioned Graph Networks for Relational Reasoning

Solving grounded language tasks often requires reasoning about relations...
research
11/19/2015

Reasoning in Vector Space: An Exploratory Study of Question Answering

Question answering tasks have shown remarkable progress with distributed...
research
10/24/2020

Learning Contextualized Knowledge Structures for Commonsense Reasoning

Recently, neural-symbolic architectures have achieved success on commons...

Please sign up or login with your details

Forgot password? Click here to reset