Visual Distant Supervision for Scene Graph Generation

by   Yuan Yao, et al.

Scene graph generation aims to identify objects and their relations in images, providing structured image representations that can facilitate numerous applications in computer vision. However, scene graph models usually require supervised learning on large quantities of labeled data with intensive human annotation. In this work, we propose visual distant supervision, a novel paradigm of visual relation learning, which can train scene graph models without any human-labeled data. The intuition is that by aligning commonsense knowledge bases and images, we can automatically create large-scale labeled data to provide distant supervision for visual relation learning. To alleviate the noise in distantly labeled data, we further propose a framework that iteratively estimates the probabilistic relation labels and eliminates the noisy ones. Comprehensive experimental results show that our distantly supervised model outperforms strong weakly supervised and semi-supervised baselines. By further incorporating human-labeled data in a semi-supervised fashion, our model outperforms state-of-the-art fully supervised models by a large margin (e.g., 8.6 micro- and 7.6 macro-recall@50 improvements for predicate classification in Visual Genome evaluation). All the data and code will be available to facilitate future research.


page 1

page 8

page 14


Self-semi-supervised Learning to Learn from NoisyLabeled Data

The remarkable success of today's deep neural networks highly depends on...

EZLearn: Exploiting Organic Supervision in Large-Scale Data Annotation

Many real-world applications require large-scale data annotation, such a...

Scene Graph Prediction with Limited Labels

Visual knowledge bases such as Visual Genome power numerous applications...

ConceptLearner: Discovering Visual Concepts from Weakly Labeled Image Collections

Discovering visual knowledge from weakly labeled data is crucial to scal...

Labeled Data Generation with Inexact Supervision

The recent advanced deep learning techniques have shown the promising re...

Semi-supervised Learning with Deterministic Labeling and Large Margin Projection

The centrality and diversity of the labeled data are very influential to...

Temporal Action Detection with Multi-level Supervision

Training temporal action detection in videos requires large amounts of l...