DeepAI AI Chat
Log In Sign Up

The Potential of Learned Index Structures for Index Compression

11/16/2018
by   Harrie Oosterhuis, et al.
University of Amsterdam
RMIT University
0

Inverted indexes are vital in providing fast key-word-based search. For every term in the document collection, a list of identifiers of documents in which the term appears is stored, along with auxiliary information such as term frequency, and position offsets. While very effective, inverted indexes have large memory requirements for web-sized collections. Recently, the concept of learned index structures was introduced, where machine learned models replace common index structures such as B-tree-indexes, hash-indexes, and bloom-filters. These learned index structures require less memory, and can be computationally much faster than their traditional counterparts. In this paper, we consider whether such models may be applied to conjunctive Boolean querying. First, we investigate how a learned model can replace document postings of an inverted index, and then evaluate the compromises such an approach might have. Second, we evaluate the potential gains that can be achieved in terms of memory requirements. Our work shows that learned models have great potential in inverted indexing, and this direction seems to be a promising area for future research.

READ FULL TEXT

page 3

page 4

11/11/2022

Efficient Immediate-Access Dynamic Indexing

In a dynamic retrieval system, documents must be ingested as they arrive...
12/04/2017

The Case for Learned Index Structures

Indexes are models: a B-Tree-Index can be seen as a model to map a key t...
06/23/2020

Benchmarking Learned Indexes

Recent advancements in learned index structures propose replacing existi...
03/01/2021

CARMI: A Cache-Aware Learned Index with a Cost-based Construction Algorithm

Learned indexes, which use machine learning models to replace traditiona...
11/29/2019

SOSD: A Benchmark for Learned Indexes

A groundswell of recent work has focused on improving data management sy...
05/11/2022

LSI: A Learned Secondary Index Structure

Learned index structures have been shown to achieve favorable lookup per...
08/11/2021

Towards Practical Learned Indexing

Latest research proposes to replace existing index structures with learn...