Log In Sign Up

Cross-Media Similarity Evaluation for Web Image Retrieval in the Wild

by   Jianfeng Dong, et al.

In order to retrieve unlabeled images by textual queries, cross-media similarity computation is a key ingredient. Although novel methods are continuously introduced, little has been done to evaluate these methods together with large-scale query log analysis. Consequently, how far have these methods brought us in answering real-user queries is unclear. Given baseline methods that compute cross-media similarity using relatively simple text/image matching, how much progress have advanced models made is also unclear. This paper takes a pragmatic approach to answering the two questions. Queries are automatically categorized according to the proposed query visualness measure, and later connected to the evaluation of multiple cross-media similarity models on three test sets. Such a connection reveals that the success of the state-of-the-art is mainly attributed to their good performance on visual-oriented queries, while these queries account for only a small part of real-user queries. To quantify the current progress, we propose a simple text2image method, representing a novel test query by a set of images selected from large-scale query log. Consequently, computing cross-media similarity between the test query and a given image boils down to comparing the visual similarity between the given image and the selected images. Image retrieval experiments on the challenging Clickture dataset show that the proposed text2image compares favorably to recent deep learning based alternatives.


page 1

page 8

page 9

page 10

page 12


Probabilistic Compositional Embeddings for Multimodal Image Retrieval

Existing works in image retrieval often consider retrieving images with ...

QAIR: Practical Query-efficient Black-Box Attacks for Image Retrieval

We study the query-based attack against image retrieval to evaluate its ...

Validating Simulations of User Query Variants

System-oriented IR evaluations are limited to rather abstract understand...

A New Evaluation Protocol and Benchmarking Results for Extendable Cross-media Retrieval

This paper proposes a new evaluation protocol for cross-media retrieval ...

Improving Calibration in Deep Metric Learning With Cross-Example Softmax

Modern image retrieval systems increasingly rely on the use of deep neur...

VRFP: On-the-fly Video Retrieval using Web Images and Fast Fisher Vector Products

VRFP is a real-time video retrieval framework based on short text input ...