Cross-Modality Attention with Semantic Graph Embedding for Multi-Label Classification

12/17/2019
by   Renchun You, et al.
0

Multi-label image and video classification are fundamental yet challenging tasks in computer vision. The main challenges lie in capturing spatial or temporal dependencies between labels and discovering the locations of discriminative features for each class. In order to overcome these challenges, we propose to use cross-modality attention with semantic graph embedding for multi label classification. Based on the constructed label graph, we propose an adjacency-based similarity graph embedding method to learn semantic label embeddings, which explicitly exploit label relationships. Then our novel cross-modality attention maps are generated with the guidance of learned label embeddings. Experiments on two multi-label image classification datasets (MS-COCO and NUS-WIDE) show our method outperforms other existing state-of-the-arts. In addition, we validate our method on a large multi-label video classification dataset (YouTube-8M Segments) and the evaluation results demonstrate the generalization capability of our method.

READ FULL TEXT

page 2

page 7

research
03/08/2022

Graph Attention Transformer Network for Multi-Label Image Classification

Multi-label classification aims to recognize multiple objects or attribu...
research
03/29/2021

Classifying Video based on Automatic Content Detection Overview

Video classification and analysis is always a popular and challenging fi...
research
07/18/2023

PatchCT: Aligning Patch Set and Label Set with Conditional Transport for Multi-Label Image Classification

Multi-label image classification is a prediction task that aims to ident...
research
11/12/2019

Pose Guided Attention for Multi-label Fashion Image Classification

We propose a compact framework with guided attention for multi-label cla...
research
03/21/2019

Semantic Comparison of State-of-the-Art Deep Learning Methods for Image Multi-Label Classification

Image understanding relies heavily on accurate multi-label classificatio...
research
12/26/2020

Coarse to Fine: Multi-label Image Classification with Global/Local Attention

In our daily life, the scenes around us are always with multiple labels ...
research
11/24/2021

Spatial-context-aware deep neural network for multi-class image classification

Multi-label image classification is a fundamental but challenging task i...

Please sign up or login with your details

Forgot password? Click here to reset