DeepAI AI Chat
Log In Sign Up

Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions

by   Marcella Cornia, et al.

Current captioning approaches can describe images using black-box architectures whose behavior is hardly controllable and explainable from the exterior. As an image can be described in infinite ways depending on the goal and the context at hand, a higher degree of controllability is needed to apply captioning algorithms in complex scenarios. In this paper, we introduce a novel framework for image captioning which can generate diverse descriptions by allowing both grounding and controllability. Given a control signal in the form of a sequence or set of image regions, we generate the corresponding caption through a recurrent architecture which predicts textual chunks explicitly grounded on regions, following the constraints of the given control. Experiments are conducted on Flickr30k Entities and on COCO Entities, an extended version of COCO in which we add grounding annotations collected in a semi-automatic manner. Results demonstrate that our method achieves state of the art performances on controllable image captioning, in terms of caption quality and diversity. Code will be made publicly available.


page 6

page 7

page 8

page 12

page 13

page 14


Controllable Image Captioning

State-of-the-art image captioners can generate accurate sentences to des...

Neural Twins Talk

Inspired by how the human brain employs more neural pathways when increa...

Human-like Controllable Image Captioning with Verb-specific Semantic Roles

Controllable Image Captioning (CIC) – generating image descriptions foll...

Controllable Image Captioning via Prompting

Despite the remarkable progress of image captioning, existing captioners...

Integrating Image Captioning with Rule-based Entity Masking

Given an image, generating its natural language description (i.e., capti...

Neural Baby Talk

We introduce a novel framework for image captioning that can produce nat...

Language-Driven Region Pointer Advancement for Controllable Image Captioning

Controllable Image Captioning is a recent sub-field in the multi-modal t...

Code Repositories


Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions. CVPR 2019

view repo