Rethinking the Two-Stage Framework for Grounded Situation Recognition

12/10/2021
by   Meng Wei, et al.
0

Grounded Situation Recognition (GSR), i.e., recognizing the salient activity (or verb) category in an image (e.g., buying) and detecting all corresponding semantic roles (e.g., agent and goods), is an essential step towards "human-like" event understanding. Since each verb is associated with a specific set of semantic roles, all existing GSR methods resort to a two-stage framework: predicting the verb in the first stage and detecting the semantic roles in the second stage. However, there are obvious drawbacks in both stages: 1) The widely-used cross-entropy (XE) loss for object recognition is insufficient in verb classification due to the large intra-class variation and high inter-class similarity among daily activities. 2) All semantic roles are detected in an autoregressive manner, which fails to model the complex semantic relations between different roles. To this end, we propose a novel SituFormer for GSR which consists of a Coarse-to-Fine Verb Model (CFVM) and a Transformer-based Noun Model (TNM). CFVM is a two-step verb prediction model: a coarse-grained model trained with XE loss first proposes a set of verb candidates, and then a fine-grained model trained with triplet loss re-ranks these candidates with enhanced verb features (not only separable but also discriminative). TNM is a transformer-based semantic role detection model, which detects all roles parallelly. Owing to the global relation modeling ability and flexibility of the transformer decoder, TNM can fully explore the statistical dependency of the roles. Extensive validations on the challenging SWiG benchmark show that SituFormer achieves a new state-of-the-art performance with significant gains under various metrics. Code is available at https://github.com/kellyiss/SituFormer.

READ FULL TEXT

page 1

page 2

page 4

page 7

research
11/19/2021

Grounded Situation Recognition with Transformers

Grounded Situation Recognition (GSR) is the task that not only classifie...
research
03/30/2022

Collaborative Transformers for Grounded Situation Recognition

Grounded situation recognition is the task of predicting the main activi...
research
07/02/2023

ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognition

Situation Recognition is the task of generating a structured summary of ...
research
07/15/2023

Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairments

Grounded Situation Recognition (GSR) is capable of recognizing and inter...
research
09/10/2019

Cross-X Learning for Fine-Grained Visual Categorization

Recognizing objects from subcategories with very subtle differences rema...
research
08/14/2017

Situation Recognition with Graph Neural Networks

We address the problem of recognizing situations in images. Given an ima...
research
06/30/2020

PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning

To build a high-quality open-domain chatbot, we introduce the effective ...

Please sign up or login with your details

Forgot password? Click here to reset