SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning

11/17/2016
by   Long Chen, et al.
0

Visual attention has been successfully applied in structural prediction tasks such as visual captioning and question answering. Existing visual attention models are generally spatial, i.e., the attention is modeled as spatial probabilities that re-weight the last conv-layer feature map of a CNN encoding an input image. However, we argue that such spatial attention does not necessarily conform to the attention mechanism --- a dynamic feature extractor that combines contextual fixations over time, as CNN features are naturally spatial, channel-wise and multi-layer. In this paper, we introduce a novel convolutional neural network dubbed SCA-CNN that incorporates Spatial and Channel-wise Attentions in a CNN. In the task of image captioning, SCA-CNN dynamically modulates the sentence generation context in multi-layer feature maps, encoding where (i.e., attentive spatial locations at multiple layers) and what (i.e., attentive channels) the visual attention is. We evaluate the proposed SCA-CNN architecture on three benchmark image captioning datasets: Flickr8K, Flickr30K, and MSCOCO. It is consistently observed that SCA-CNN significantly outperforms state-of-the-art visual attention-based image captioning methods.

READ FULL TEXT

page 1

page 8

research
11/29/2021

Neural Attention for Image Captioning: Review of Outstanding Methods

Image captioning is the task of automatically generating sentences that ...
research
06/08/2016

Progressive Attention Networks for Visual Attribute Prediction

We propose a novel attention model that can accurately attend to target ...
research
11/23/2016

Adaptive Feature Abstraction for Translating Video to Text

Previous models for video captioning often use the output from a specifi...
research
03/31/2020

X-Linear Attention Networks for Image Captioning

Recent progress on fine-grained visual recognition and visual question a...
research
06/16/2019

Image Captioning with Integrated Bottom-Up and Multi-level Residual Top-Down Attention for Game Scene Understanding

Image captioning has attracted considerable attention in recent years. H...
research
11/02/2020

Dual Attention on Pyramid Feature Maps for Image Captioning

Generating natural sentences from images is a fundamental learning task ...
research
02/23/2017

ViP-CNN: Visual Phrase Guided Convolutional Neural Network

As the intermediate level task connecting image captioning and object de...

Please sign up or login with your details

Forgot password? Click here to reset