Joint Learning of Deep Retrieval Model and Product Quantization based Embedding Index

05/09/2021
by   Han Zhang, et al.
0

Embedding index that enables fast approximate nearest neighbor(ANN) search, serves as an indispensable component for state-of-the-art deep retrieval systems. Traditional approaches, often separating the two steps of embedding learning and index building, incur additional indexing time and decayed retrieval accuracy. In this paper, we propose a novel method called Poeem, which stands for product quantization based embedding index jointly trained with deep retrieval model, to unify the two separate steps within an end-to-end training, by utilizing a few techniques including the gradient straight-through estimator, warm start strategy, optimal space decomposition and Givens rotation. Extensive experimental results show that the proposed method not only improves retrieval accuracy significantly but also reduces the indexing time to almost none. We have open sourced our approach for the sake of comparison and reproducibility.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

03/15/2016

Scalable Image Retrieval by Sparse Product Quantization

Fast Approximate Nearest Neighbor (ANN) search technique for high-dimens...
11/19/2018

End-to-End Retrieval in Continuous Space

Most text-based information retrieval (IR) systems index objects by word...
08/02/2021

Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance

Recently, Information Retrieval community has witnessed fast-paced advan...
09/29/2020

SIR: Similar Image Retrieval for Product Search in E-Commerce

We present a similar image retrieval (SIR) platform that is used to quic...
01/02/2019

Vector and Line Quantization for Billion-scale Similarity Search on GPUs

Billion-scale high-dimensional approximate nearest neighbour (ANN) searc...
11/23/2017

In Defense of Product Quantization

Despite their widespread adoption, Product Quantization techniques were ...
06/15/2018

Efficient Nearest Neighbors Search for Large-Scale Landmark Recognition

The problem of landmark recognition has achieved excellent results in sm...

Code Repositories

poeem

A library for end-to-end learning of embedding index and retrieval model


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.