Triplet-Aware Scene Graph Embeddings

by   Brigit Schroeder, et al.
University of California Santa Cruz

Scene graphs have become an important form of structured knowledge for tasks such as for image generation, visual relation detection, visual question answering, and image retrieval. While visualizing and interpreting word embeddings is well understood, scene graph embeddings have not been fully explored. In this work, we train scene graph embeddings in a layout generation task with different forms of supervision, specifically introducing triplet super-vision and data augmentation. We see a significant performance increase in both metrics that measure the goodness of layout prediction, mean intersection-over-union (mIoU)(52.3 54.1 understand how these different methods affect the scene graph representation, we apply several new visualization and evaluation methods to explore the evolution of the scene graph embedding. We find that triplet supervision significantly improves the embedding separability, which is highly correlated with the performance of the layout prediction model.


page 3

page 4


Scene Graph Embeddings Using Relative Similarity Supervision

Scene graphs are a powerful structured representation of the underlying ...

Scene Graph to Image Generation with Contextualized Object Layout Refinement

Generating high-quality images from scene graphs, that is, graphs that d...

Compact Scene Graphs for Layout Composition and Patch Retrieval

Structured representations such as scene graphs serve as an efficient an...

Image Semantic Relation Generation

Scene graphs provide structured semantic understanding beyond images. Fo...

Learning Similarity between Scene Graphs and Images with Transformers

Scene graph generation is conventionally evaluated by (mean) Recall@K, w...

Accurate and Scalable Version Identification Using Musically-Motivated Embeddings

The version identification (VI) task deals with the automatic detection ...

Please sign up or login with your details

Forgot password? Click here to reset