Image Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 Challenge

12/21/2020
by   Pierre Dognin, et al.
6

Image captioning has recently demonstrated impressive progress largely owing to the introduction of neural network algorithms trained on curated dataset like MS-COCO. Often work in this field is motivated by the promise of deployment of captioning systems in practical applications. However, the scarcity of data and contexts in many competition datasets renders the utility of systems trained on these datasets limited as an assistive technology in real-world settings, such as helping visually impaired people navigate and accomplish everyday tasks. This gap motivated the introduction of the novel VizWiz dataset, which consists of images taken by the visually impaired and captions that have useful, task-oriented information. In an attempt to help the machine learning computer vision field realize its promise of producing technologies that have positive social impact, the curators of the VizWiz dataset host several competitions, including one for image captioning. This work details the theory and engineering from our winning submission to the 2020 captioning competition. Our work provides a step towards improved assistive image captioning systems.

READ FULL TEXT

page 2

page 3

page 4

page 9

page 10

research
08/05/2023

A Comprehensive Analysis of Real-World Image Captioning and Scene Identification

Image captioning is a computer vision task that involves generating natu...
research
12/21/2020

Alleviating Noisy Data in Image Captioning with Cooperative Distillation

Image captioning systems have made substantial progress, largely due to ...
research
02/20/2020

Captioning Images Taken by People Who Are Blind

While an important problem in the vision community is to design algorith...
research
03/21/2021

#PraCegoVer: A Large Dataset for Image Captioning in Portuguese

Automatically describing images using natural sentences is an important ...
research
02/11/2022

Bench-Marking And Improving Arabic Automatic Image Captioning Through The Use Of Multi-Task Learning Paradigm

The continuous increase in the use of social media and the visual conten...
research
06/14/2022

Automated Testing of Image Captioning Systems

Image captioning (IC) systems, which automatically generate a text descr...

Please sign up or login with your details

Forgot password? Click here to reset