SD-RSIC: Summarization Driven Deep Remote Sensing Image Captioning

06/15/2020
by   Gencer Sumbul, et al.
0

Deep neural networks (DNNs) have been recently found popular for image captioning problems in remote sensing (RS). Existing DNN based approaches rely on the availability of a training set made up of a high number of RS images with their captions. However, captions of training images may contain redundant information (they can be repetitive or semantically similar to each other), resulting in information deficiency while learning a mapping from image domain to language domain. To overcome this limitation, in this paper we present a novel Summarization Driven Remote Sensing Image Captioning (SD-RSIC) approach. The proposed approach consists of three main steps. The first step obtains the standard image captions by jointly exploiting convolutional neural networks (CNNs) with long short-term memory (LSTM) networks. The second step, unlike the existing RS image captioning methods, summarizes the ground-truth captions of each training image into a single caption by exploiting sequence to sequence neural networks and eliminates the redundancy present in the training set. The third step automatically defines the adaptive weights associated to each RS image to combine the standard captions with the summarized captions based on the semantic content of the image. This is achieved by a novel adaptive weighting strategy defined in the context of LSTM networks. Experimental results obtained on the RSCID, UCM-Captions and Sydney-Captions datasets show the effectiveness of the proposed approach compared to the state-of-the-art RS image captioning approaches.

READ FULL TEXT

page 1

page 3

page 5

page 10

research
10/05/2020

A Novel Actor Dual-Critic Model for Remote Sensing Image Captioning

We deal with the problem of generating textual captions from optical rem...
research
07/10/2018

"Factual" or "Emotional": Stylized Image Captioning with Adaptive Learning and Attention

Generating stylized captions for an image is an emerging topic in image ...
research
04/08/2022

On Distinctive Image Captioning via Comparing and Reweighting

Recent image captioning models are achieving impressive results based on...
research
06/20/2020

Remote Sensing Image Scene Classification with Deep Neural Networks in JPEG 2000 Compressed Domain

To reduce the storage requirements, remote sensing (RS) images are usual...
research
07/25/2018

Distinctive-attribute Extraction for Image Captioning

Image captioning, an open research issue, has been evolved with the prog...
research
04/19/2021

A SAR speckle filter based on Residual Convolutional Neural Networks

In recent years, Machine Learning (ML) algorithms have become widespread...
research
06/24/2021

Domain-guided Machine Learning for Remotely Sensed In-Season Crop Growth Estimation

Advanced machine learning techniques have been used in remote sensing (R...

Please sign up or login with your details

Forgot password? Click here to reset