Natural Language Guided Visual Relationship Detection

11/16/2017
by   Wentong Liao, et al.
1

Reasoning about the relationships between object pairs in images is a crucial task for holistic scene understanding. Most of the existing works treat this task as a pure visual classification task: each type of relationship or phrase is classified as a relation category based on the extracted visual features. However, each kind of relationships has a wide variety of object combination and each pair of objects has diverse interactions. Obtaining sufficient training samples for all possible relationship categories is difficult and expensive. In this work, we propose a natural language guided framework to tackle this problem. We propose to use a generic bi-directional recurrent neural network to predict the semantic connection between the participating objects in the relationship from the aspect of natural language. The proposed simple method achieves the state-of-the-art on the Visual Relationship Detection (VRD) and Visual Genome datasets, especially when predicting unseen relationships (e.g. recall improved from 76.42 testing set).

READ FULL TEXT

page 1

page 2

page 3

page 5

page 6

page 7

page 10

page 11

research
04/03/2019

Exploring the Semantics for Visual Relationship Detection

Scene graph construction / visual relationship detection from an image a...
research
07/31/2016

Visual Relationship Detection with Language Priors

Visual relationships capture a wide variety of interactions between pair...
research
04/11/2017

Detecting Visual Relationships with Deep Relational Networks

Relationships among objects play a crucial role in image understanding. ...
research
02/23/2017

ViP-CNN: Visual Phrase Guided Convolutional Neural Network

As the intermediate level task connecting image captioning and object de...
research
05/28/2018

Visual Relationship Detection Based on Guided Proposals and Semantic Knowledge Distillation

A thorough comprehension of image content demands a complex grasp of the...
research
10/11/2020

Constructing a Visual Relationship Authenticity Dataset

A visual relationship denotes a relationship between two objects in an i...
research
09/11/2018

Context-Dependent Diffusion Network for Visual Relationship Detection

Visual relationship detection can bridge the gap between computer vision...

Please sign up or login with your details

Forgot password? Click here to reset