DeepAI AI Chat
Log In Sign Up

Situation Recognition with Graph Neural Networks

by   Ruiyu Li, et al.
The Chinese University of Hong Kong

We address the problem of recognizing situations in images. Given an image, the task is to predict the most salient verb (action), and fill its semantic roles such as who is performing the action, what is the source and target of the action, etc. Different verbs have different roles (e.g. attacking has weapon), and each role can take on many possible values (nouns). We propose a model based on Graph Neural Networks that allows us to efficiently capture joint dependencies between roles using neural networks defined on a graph. Experiments with different graph connectivities show that our approach that propagates information between roles significantly outperforms existing work, as well as multiple baselines. We obtain roughly 3-5 work in predicting the full situation. We also provide a thorough qualitative analysis of our model and influence of different roles in the verbs.


page 1

page 2

page 4

page 6

page 7

page 11

page 12

page 22


CopulaGNN: Towards Integrating Representational and Correlational Roles of Graphs in Graph Neural Networks

Graph-structured data are ubiquitous. However, graphs encode diverse typ...

Continuous Graph Neural Networks

This paper builds the connection between graph neural networks and tradi...

Revisiting Role Discovery in Networks: From Node to Edge Roles

Previous work in network analysis has focused on modeling the mixed-memb...

Rethinking the Two-Stage Framework for Grounded Situation Recognition

Grounded Situation Recognition (GSR), i.e., recognizing the salient acti...

Graph Neural Networks in Real-Time Fraud Detection with Lambda Architecture

Transaction checkout fraud detection is an essential risk control compon...

A Method to Predict Semantic Relations on Artificial Intelligence Papers

Predicting the emergence of links in large evolving networks is a diffic...

Measuring Thematic Fit with Distributional Feature Overlap

In this paper, we introduce a new distributional method for modeling pre...