Improving Reinforcement Learning Based Image Captioning with Natural Language Prior

09/13/2018
by   Tszhang Guo, et al.
0

Recently, Reinforcement Learning (RL) approaches have demonstrated advanced performance in image captioning by directly optimizing the metric used for testing. However, this shaped reward introduces learning biases, which reduces the readability of generated text. In addition, the large sample space makes training unstable and slow. To alleviate these issues, we propose a simple coherent solution that constrains the action space using an n-gram language prior. Quantitative and qualitative evaluations on benchmarks show that RL with the simple add-on module performs favorably against its counterpart in terms of both readability and speed of convergence. Human evaluation results show that our model is more human readable and graceful. The implementation will become publicly available upon the acceptance of the paper.

READ FULL TEXT
research
05/18/2018

Improving Image Captioning with Conditional Generative Adversarial Nets

In this paper, we propose a novel conditional generative adversarial net...
research
06/29/2017

Actor-Critic Sequence Training for Image Captioning

Generating natural language descriptions of images is an important capab...
research
08/16/2018

Context-Aware Visual Policy Network for Sequence-Level Image Captioning

Many vision-language tasks can be reduced to the problem of sequence pre...
research
05/24/2023

Gender Biases in Automatic Evaluation Metrics: A Case Study on Image Captioning

Pretrained model-based evaluation metrics have demonstrated strong perfo...
research
02/14/2020

ResCap V1: Deep Residual Learning Based Image Captioning

Image Captioning alludes to the process of generating text description f...
research
04/08/2022

Semantic Exploration from Language Abstractions and Pretrained Representations

Continuous first-person 3D environments pose unique exploration challeng...
research
09/05/2018

Accelerated Reinforcement Learning for Sentence Generation by Vocabulary Prediction

A major obstacle in reinforcement learning-based sentence generation is ...

Please sign up or login with your details

Forgot password? Click here to reset