DeepAI AI Chat
Log In Sign Up

Polysemous codes

09/07/2016
by   Matthijs Douze, et al.
0

This paper considers the problem of approximate nearest neighbor search in the compressed domain. We introduce polysemous codes, which offer both the distance estimation quality of product quantization and the efficient comparison of binary codes with Hamming distance. Their design is inspired by algorithms introduced in the 90's to construct channel-optimized vector quantizers. At search time, this dual interpretation accelerates the search. Most of the indexed vectors are filtered out with Hamming distance, letting only a fraction of the vectors to be ranked with an asymmetric distance estimator. The method is complementary with a coarse partitioning of the feature space such as the inverted multi-index. This is shown by our experiments performed on several public benchmarks such as the BIGANN dataset comprising one billion vectors, for which we report state-of-the-art results for query times below 0.3 millisecond per core. Last but not least, our approach allows the approximate computation of the k-NN graph associated with the Yahoo Flickr Creative Commons 100M, described by CNN image descriptors, in less than 8 hours on a single machine.

READ FULL TEXT
09/18/2020

Accelerating Search on Binary Codes in Weighted Hamming Space

Compared to Hamming distance, weighted Hamming distance as a similarity ...
04/24/2017

Accelerated Nearest Neighbor Search with Quick ADC

Efficient Nearest Neighbor (NN) search in high-dimensional spaces is a f...
06/30/2017

Bolt: Accelerated Data Mining with Fast Vector Compression

Vectors of data are at the heart of machine learning and data mining. Re...
05/16/2019

Derived Codebooks for High-Accuracy Nearest Neighbor Search

High-dimensional Nearest Neighbor (NN) search is central in multimedia s...
12/17/2021

Nearest neighbor search with compact codes: A decoder perspective

Modern approaches for fast retrieval of similar vectors on billion-scale...
02/28/2017

Billion-scale similarity search with GPUs

Similarity search finds application in specialized database systems hand...
03/25/2019

Local Orthogonal Decomposition for Maximum Inner Product Search

Inverted file and asymmetric distance computation (IVFADC) have been suc...

Code Repositories

faiss

A library for efficient similarity search and clustering of dense vectors.


view repo

faiss

A library for efficient similarity search and clustering of dense vectors.


view repo

faiss

A library for efficient similarity search and clustering of dense vectors.


view repo

faiss

A library for efficient similarity search and clustering of dense vectors.


view repo