Energy-Based Learning for Scene Graph Generation

03/03/2021
by   Mohammed Suhail, et al.
0

Traditional scene graph generation methods are trained using cross-entropy losses that treat objects and relationships as independent entities. Such a formulation, however, ignores the structure in the output space, in an inherently structured prediction problem. In this work, we introduce a novel energy-based learning framework for generating scene graphs. The proposed formulation allows for efficiently incorporating the structure of scene graphs in the output space. This additional constraint in the learning framework acts as an inductive bias and allows models to learn efficiently from a small number of labels. We use the proposed energy-based framework to train existing state-of-the-art models and obtain a significant performance improvement, of up to 21 Furthermore, we showcase the learning efficiency of the proposed framework by demonstrating superior performance in the zero- and few-shot settings where data is scarce.

READ FULL TEXT

page 1

page 4

page 8

research
11/30/2022

Iterative Scene Graph Generation with Generative Transformers

Scene graphs provide a rich, structured representation of a scene by enc...
research
05/17/2020

Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation

Scene graph generation (SGG) aims to predict graph-structured descriptio...
research
05/10/2023

Incorporating Structured Representations into Pretrained Vision Language Models Using Scene Graphs

Vision and Language (VL) models have demonstrated remarkable zero-shot p...
research
06/12/2019

Visual Relationships as Functions: Enabling Few-Shot Scene Graph Prediction

Scene graph prediction --- classifying the set of objects and predicates...
research
09/29/2022

Prompt-guided Scene Generation for 3D Zero-Shot Learning

Zero-shot learning on 3D point cloud data is a related underexplored pro...
research
09/07/2023

Zero-Shot Scene Graph Generation via Triplet Calibration and Reduction

Scene Graph Generation (SGG) plays a pivotal role in downstream vision-l...
research
05/26/2023

Automatic Tuning of Loss Trade-offs without Hyper-parameter Search in End-to-End Zero-Shot Speech Synthesis

Recently, zero-shot TTS and VC methods have gained attention due to thei...

Please sign up or login with your details

Forgot password? Click here to reset