Composing Text and Image for Image Retrieval - An Empirical Odyssey

by   Nam Vo, et al.

In this paper, we study the task of image retrieval, where the input query is specified in the form of an image plus some text that describes desired modifications to the input image. For example, we may present an image of the Eiffel tower, and ask the system to find images which are visually similar but are modified in small ways, such as being taken at nighttime instead of during the day. To tackle this task, we learn a similarity metric between a target image and a source image plus source text, an embedding and composing function such that target image feature is close to the source image plus text composition feature. We propose a new way to combine image and text using such function that is designed for the retrieval task. We show this outperforms existing approaches on 3 different datasets, namely Fashion-200k, MIT-States and a new synthetic dataset we create based on CLEVR. We also show that our approach can be used to classify input queries, in addition to image retrieval.


page 1

page 3

page 6

page 7

page 8


Probabilistic Compositional Embeddings for Multimodal Image Retrieval

Existing works in image retrieval often consider retrieving images with ...

Using Text to Teach Image Retrieval

Image retrieval relies heavily on the quality of the data modeling and t...

ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity

An intuitive way to search for images is to use queries composed of an e...

Designovel's system description for Fashion-IQ challenge 2019

This paper describes Designovel's systems which are submitted to the Fas...

Text as Neural Operator: Image Manipulation by Text Instruction

In this paper, we study a new task that allows users to edit an input im...

Compositional Learning of Image-Text Query for Image Retrieval

In this paper, we investigate the problem of retrieving images from a da...

RTIC: Residual Learning for Text and Image Composition using Graph Convolutional Network

In this paper, we study the compositional learning of images and texts f...

Code Repositories


deep learning, image retrieval, vision and language

view repo


Modality-Agnostic Attention Fusion for visual search with text feedback

view repo


The project is based on an image and text retrieval system

view repo


To implement the idea of using scene-graphs for text based image retrieval.

view repo