Resolving Referring Expressions in Images With Labeled Elements

10/24/2018
by   Nevan Wichers, et al.
0

Images may have elements containing text and a bounding box associated with them, for example, text identified via optical character recognition on a computer screen image, or a natural image with labeled objects. We present an end-to-end trainable architecture to incorporate the information from these elements and the image to segment/identify the part of the image a natural language expression is referring to. We calculate an embedding for each element and then project it onto the corresponding location (i.e., the associated bounding box) of the image feature map. We show that this architecture gives an improvement in resolving referring expressions, over only using the image, and other methods that incorporate the element information. We demonstrate experimental results on the referring expression datasets based on COCO, and on a webpage image referring expression dataset that we developed.

READ FULL TEXT

page 2

page 4

page 6

research
01/26/2016

COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images

This paper describes the COCO-Text dataset. In recent years large-scale ...
research
01/24/2018

MAttNet: Modular Attention Network for Referring Expression Comprehension

In this paper, we address referring expression comprehension: localizing...
research
04/05/2022

Text Spotting Transformers

In this paper, we present TExt Spotting TRansformers (TESTR), a generic ...
research
09/02/2018

Natural Language Person Search Using Deep Reinforcement Learning

Recent success in deep reinforcement learning is having an agent learn h...
research
03/22/2017

An End-to-End Approach to Natural Language Object Retrieval via Context-Aware Deep Reinforcement Learning

We propose an end-to-end approach to the natural language object retriev...
research
03/22/2022

Open-Vocabulary DETR with Conditional Matching

Open-vocabulary object detection, which is concerned with the problem of...
research
05/05/2021

Proposal-free One-stage Referring Expression via Grid-Word Cross-Attention

Referring Expression Comprehension (REC) has become one of the most impo...

Please sign up or login with your details

Forgot password? Click here to reset