Aesthetically Relevant Image Captioning

11/25/2022
by   Zhipeng Zhong, et al.
0

Image aesthetic quality assessment (AQA) aims to assign numerical aesthetic ratings to images whilst image aesthetic captioning (IAC) aims to generate textual descriptions of the aesthetic aspects of images. In this paper, we study image AQA and IAC together and present a new IAC method termed Aesthetically Relevant Image Captioning (ARIC). Based on the observation that most textual comments of an image are about objects and their interactions rather than aspects of aesthetics, we first introduce the concept of Aesthetic Relevance Score (ARS) of a sentence and have developed a model to automatically label a sentence with its ARS. We then use the ARS to design the ARIC model which includes an ARS weighted IAC loss function and an ARS based diverse aesthetic caption selector (DACS). We present extensive experimental results to show the soundness of the ARS concept and the effectiveness of the ARIC model by demonstrating that texts with higher ARS's can predict the aesthetic ratings more accurately and that the new ARIC model can generate more accurate, aesthetically more relevant and more diverse image captions. Furthermore, a large new research database containing 510K images with over 5 million comments and 350K aesthetic scores, and code for implementing ARIC are available at https://github.com/PengZai/ARIC.

READ FULL TEXT

page 7

page 18

page 19

page 20

research
07/12/2022

A Baseline for Detecting Out-of-Distribution Examples in Image Captioning

Image captioning research achieved breakthroughs in recent years by deve...
research
08/04/2021

ICECAP: Information Concentrated Entity-aware Image Captioning

Most current image captioning systems focus on describing general image ...
research
08/19/2014

What makes an Image Iconic? A Fine-Grained Case Study

A natural approach to teaching a visual concept, e.g. a bird species, is...
research
04/27/2022

CapOnImage: Context-driven Dense-Captioning on Image

Existing image captioning systems are dedicated to generating narrative ...
research
11/07/2015

Generation and Comprehension of Unambiguous Object Descriptions

We propose a method that can generate an unambiguous description (known ...
research
07/11/2019

Aesthetic Attributes Assessment of Images

Image aesthetic quality assessment has been a relatively hot topic durin...
research
03/06/2020

Captioning Images with Novel Objects via Online Vocabulary Expansion

In this study, we introduce a low cost method for generating description...

Please sign up or login with your details

Forgot password? Click here to reset