DeepAI AI Chat
Log In Sign Up

Visual Relationships as Functions: Enabling Few-Shot Scene Graph Prediction

by   Apoorva Dornadula, et al.
Stanford University

Scene graph prediction --- classifying the set of objects and predicates in a visual scene --- requires substantial training data. The long-tailed distribution of relationships can be an obstacle for such approaches, however, as they can only be trained on the small set of predicates that carry sufficient labels. We introduce the first scene graph prediction model that supports few-shot learning of predicates, enabling scene graph approaches to generalize to a set of new predicates. First, we introduce a new model of predicates as functions that operate on object features or image locations. Next, we define a scene graph model where these functions are trained as message passing protocols within a new graph convolution framework. We train the framework with a frequently occurring set of predicates and show that our approach outperforms those that use the same amount of supervision by 1.78 at recall@50 and performs on par with other scene graph models. Next, we extract object representations generated by the trained predicate functions to train few-shot predicate classifiers on rare predicates with as few as 1 labeled example. When compared to strong baselines like transfer learning from existing state-of-the-art representations, we show improved 5-shot performance by 4.16 recall@1. Finally, we show that our predicate functions generate interpretable visualizations, enabling the first interpretable scene graph model.


page 6

page 8

page 13

page 14


Scene Graph Prediction with Limited Labels

Visual knowledge bases such as Visual Genome power numerous applications...

Scene Graph Generation for Better Image Captioning?

We investigate the incorporation of visual relationships into the task o...

One-shot Scene Graph Generation

As a structured representation of the image content, the visual scene gr...

SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentation

In this paper we propose a neural message passing approach to augment an...

Energy-Based Learning for Scene Graph Generation

Traditional scene graph generation methods are trained using cross-entro...

GPS-Net: Graph Property Sensing Network for Scene Graph Generation

Scene graph generation (SGG) aims to detect objects in an image along wi...

Generative Graph Perturbations for Scene Graph Prediction

Inferring objects and their relationships from an image is useful in man...