Log In Sign Up

Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generation

by   Gengcong Yang, et al.

To generate "accurate" scene graphs, almost all existing methods predict pairwise relationships in a deterministic manner. However, we argue that visual relationships are often semantically ambiguous. Specifically, inspired by linguistic knowledge, we classify the ambiguity into three types: Synonymy Ambiguity, Hyponymy Ambiguity, and Multi-view Ambiguity. The ambiguity naturally leads to the issue of implicit multi-label, motivating the need for diverse predictions. In this work, we propose a novel plug-and-play Probabilistic Uncertainty Modeling (PUM) module. It models each union region as a Gaussian distribution, whose variance measures the uncertainty of the corresponding visual content. Compared to the conventional deterministic methods, such uncertainty modeling brings stochasticity of feature representation, which naturally enables diverse predictions. As a byproduct, PUM also manages to cover more fine-grained relationships and thus alleviates the issue of bias towards frequent relationships. Extensive experiments on the large-scale Visual Genome benchmark show that combining PUM with newly proposed ResCAGCN can achieve state-of-the-art performances, especially under the mean recall metric. Furthermore, we prove the universal effectiveness of PUM by plugging it into some existing models and provide insightful analysis of its ability to generate diverse yet plausible visual relationships.


page 1

page 3

page 8


Knowledge-Embedded Routing Network for Scene Graph Generation

To understand a scene in depth not only involves locating/recognizing in...

Qualitative Measures of Ambiguity

This paper introduces a qualitative measure of ambiguity and analyses it...

Probabilistic Debiasing of Scene Graphs

The quality of scene graphs generated by the state-of-the-art (SOTA) mod...

Rethinking Visual Relationships for High-level Image Understanding

Relationships, as the bond of isolated entities in images, reflect the i...

AE TextSpotter: Learning Visual and Linguistic Representation for Ambiguous Text Spotting

Scene text spotting aims to detect and recognize the entire word or sent...

VGStore: A Multimodal Extension to SPARQL for Querying RDF Scene Graph

Semantic Web technology has successfully facilitated many RDF models wit...

Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding

Temporal grounding aims to localize temporal boundaries within untrimmed...