VQA with Cascade of Self- and Co-Attention Blocks

02/28/2023

∙

The use of complex attention modules has improved the performance of the Visual Question Answering (VQA) task. This work aims to learn an improved multi-modal representation through dense interaction of visual and textual modalities. The proposed model has an attention block containing both self-attention and co-attention on image and text. The self-attention modules provide the contextual information of objects (for an image) and words (for a question) that are crucial for inferring an answer. On the other hand, co-attention aids the interaction of image and text. Further, fine-grained information is obtained from two modalities by using a Cascade of Self- and Co-Attention blocks (CSCA). This proposal is benchmarked on the widely used VQA2.0 and TDIUC datasets. The efficacy of key components of the model and cascading of attention modules are demonstrated by experiments involving ablation analysis.

READ FULL TEXT

VQA with Cascade of Self- and Co-Attention Blocks

Multimodal Integration of Human-Like Attention in Visual Question Answering

A Dual-Attention Learning Network with Word and Sentence Embedding for Medical Visual Question Answering

Multimodal grid features and cell pointers for Scene Text Visual Question Answering

Modulated Self-attention Convolutional Network for VQA

EAML: Ensemble Self-Attention-based Mutual Learning Network for Document Image Classification

Medical visual question answering using joint self-supervised learning

Local Attention Graph-based Transformer for Multi-target Genetic Alteration Prediction

VQA with Cascade of Self- and Co-Attention Blocks

Related Research

Multimodal Integration of Human-Like Attention in Visual Question Answering

A Dual-Attention Learning Network with Word and Sentence Embedding for Medical Visual Question Answering

Multimodal grid features and cell pointers for Scene Text Visual Question Answering

Modulated Self-attention Convolutional Network for VQA

EAML: Ensemble Self-Attention-based Mutual Learning Network for Document Image Classification

Medical visual question answering using joint self-supervised learning

Local Attention Graph-based Transformer for Multi-target Genetic Alteration Prediction