DeepAI AI Chat
Log In Sign Up

The Potential of Learned Index Structures for Index Compression

by   Harrie Oosterhuis, et al.
University of Amsterdam
RMIT University

Inverted indexes are vital in providing fast key-word-based search. For every term in the document collection, a list of identifiers of documents in which the term appears is stored, along with auxiliary information such as term frequency, and position offsets. While very effective, inverted indexes have large memory requirements for web-sized collections. Recently, the concept of learned index structures was introduced, where machine learned models replace common index structures such as B-tree-indexes, hash-indexes, and bloom-filters. These learned index structures require less memory, and can be computationally much faster than their traditional counterparts. In this paper, we consider whether such models may be applied to conjunctive Boolean querying. First, we investigate how a learned model can replace document postings of an inverted index, and then evaluate the compromises such an approach might have. Second, we evaluate the potential gains that can be achieved in terms of memory requirements. Our work shows that learned models have great potential in inverted indexing, and this direction seems to be a promising area for future research.


page 3

page 4


Efficient Immediate-Access Dynamic Indexing

In a dynamic retrieval system, documents must be ingested as they arrive...

The Case for Learned Index Structures

Indexes are models: a B-Tree-Index can be seen as a model to map a key t...

Benchmarking Learned Indexes

Recent advancements in learned index structures propose replacing existi...

CARMI: A Cache-Aware Learned Index with a Cost-based Construction Algorithm

Learned indexes, which use machine learning models to replace traditiona...

SOSD: A Benchmark for Learned Indexes

A groundswell of recent work has focused on improving data management sy...

LSI: A Learned Secondary Index Structure

Learned index structures have been shown to achieve favorable lookup per...

Towards Practical Learned Indexing

Latest research proposes to replace existing index structures with learn...