Composing Text and Image for Image Retrieval - An Empirical Odyssey

12/18/2018
by   Nam Vo, et al.
21

In this paper, we study the task of image retrieval, where the input query is specified in the form of an image plus some text that describes desired modifications to the input image. For example, we may present an image of the Eiffel tower, and ask the system to find images which are visually similar but are modified in small ways, such as being taken at nighttime instead of during the day. To tackle this task, we learn a similarity metric between a target image and a source image plus source text, an embedding and composing function such that target image feature is close to the source image plus text composition feature. We propose a new way to combine image and text using such function that is designed for the retrieval task. We show this outperforms existing approaches on 3 different datasets, namely Fashion-200k, MIT-States and a new synthetic dataset we create based on CLEVR. We also show that our approach can be used to classify input queries, in addition to image retrieval.

READ FULL TEXT

page 1

page 3

page 6

page 7

page 8

04/12/2022

Probabilistic Compositional Embeddings for Multimodal Image Retrieval

Existing works in image retrieval often consider retrieving images with ...
11/19/2020

Using Text to Teach Image Retrieval

Image retrieval relies heavily on the quality of the data modeling and t...
03/15/2022

ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity

An intuitive way to search for images is to use queries composed of an e...
10/21/2019

Designovel's system description for Fashion-IQ challenge 2019

This paper describes Designovel's systems which are submitted to the Fas...
08/11/2020

Text as Neural Operator: Image Manipulation by Text Instruction

In this paper, we study a new task that allows users to edit an input im...
06/19/2020

Compositional Learning of Image-Text Query for Image Retrieval

In this paper, we investigate the problem of retrieving images from a da...
04/07/2021

RTIC: Residual Learning for Text and Image Composition using Graph Convolutional Network

In this paper, we study the compositional learning of images and texts f...

Code Repositories

tirg

deep learning, image retrieval, vision and language


view repo

maaf

Modality-Agnostic Attention Fusion for visual search with text feedback


view repo

image_text_search

The project is based on an image and text retrieval system


view repo

scene-graph-image-retrieval

To implement the idea of using scene-graphs for text based image retrieval.


view repo