From General to Specific: Informative Scene Graph Generation via Balance Adjustment

08/30/2021
by   Yuyu Guo, et al.
2

The scene graph generation (SGG) task aims to detect visual relationship triplets, i.e., subject, predicate, object, in an image, providing a structural vision layout for scene understanding. However, current models are stuck in common predicates, e.g., "on" and "at", rather than informative ones, e.g., "standing on" and "looking at", resulting in the loss of precise information and overall performance. If a model only uses "stone on road" rather than "blocking" to describe an image, it is easy to misunderstand the scene. We argue that this phenomenon is caused by two key imbalances between informative predicates and common ones, i.e., semantic space level imbalance and training sample level imbalance. To tackle this problem, we propose BA-SGG, a simple yet effective SGG framework based on balance adjustment but not the conventional distribution fitting. It integrates two components: Semantic Adjustment (SA) and Balanced Predicate Learning (BPL), respectively for adjusting these imbalances. Benefited from the model-agnostic process, our method is easily applied to the state-of-the-art SGG models and significantly improves the SGG performance. Our method achieves 14.3 than that of the Transformer model at three scene graph generation sub-tasks on Visual Genome, respectively. Codes are publicly available.

READ FULL TEXT

page 1

page 2

page 6

page 8

research
08/10/2023

Informative Scene Graph Generation via Debiasing

Scene graph generation aims to detect visual relationship triplets, (sub...
research
11/09/2022

SG-Shuffle: Multi-aspect Shuffle Transformer for Scene Graph Generation

Scene Graph Generation (SGG) serves a comprehensive representation of th...
research
08/09/2023

Generalized Unbiased Scene Graph Generation

Existing Unbiased Scene Graph Generation (USGG) methods only focus on ad...
research
04/04/2019

Libra R-CNN: Towards Balanced Learning for Object Detection

Compared with model architectures, the training process, which is also c...
research
03/17/2022

Biasing Like Human: A Cognitive Bias Framework for Scene Graph Generation

Scene graph generation is a sophisticated task because there is no speci...
research
06/23/2023

Towards Unseen Triples: Effective Text-Image-joint Learning for Scene Graph Generation

Scene Graph Generation (SGG) aims to structurally and comprehensively re...
research
10/20/2022

Content-based Graph Privacy Advisor

People may be unaware of the privacy risks of uploading an image online....

Please sign up or login with your details

Forgot password? Click here to reset