The Potential of Learned Index Structures for Index Compression

11/16/2018
by   Harrie Oosterhuis, et al.
0

Inverted indexes are vital in providing fast key-word-based search. For every term in the document collection, a list of identifiers of documents in which the term appears is stored, along with auxiliary information such as term frequency, and position offsets. While very effective, inverted indexes have large memory requirements for web-sized collections. Recently, the concept of learned index structures was introduced, where machine learned models replace common index structures such as B-tree-indexes, hash-indexes, and bloom-filters. These learned index structures require less memory, and can be computationally much faster than their traditional counterparts. In this paper, we consider whether such models may be applied to conjunctive Boolean querying. First, we investigate how a learned model can replace document postings of an inverted index, and then evaluate the compromises such an approach might have. Second, we evaluate the potential gains that can be achieved in terms of memory requirements. Our work shows that learned models have great potential in inverted indexing, and this direction seems to be a promising area for future research.

READ FULL TEXT

page 3

page 4

research
11/11/2022

Efficient Immediate-Access Dynamic Indexing

In a dynamic retrieval system, documents must be ingested as they arrive...
research
12/04/2017

The Case for Learned Index Structures

Indexes are models: a B-Tree-Index can be seen as a model to map a key t...
research
06/23/2020

Benchmarking Learned Indexes

Recent advancements in learned index structures propose replacing existi...
research
11/29/2019

SOSD: A Benchmark for Learned Indexes

A groundswell of recent work has focused on improving data management sy...
research
03/01/2021

CARMI: A Cache-Aware Learned Index with a Cost-based Construction Algorithm

Learned indexes, which use machine learning models to replace traditiona...
research
05/11/2022

LSI: A Learned Secondary Index Structure

Learned index structures have been shown to achieve favorable lookup per...
research
08/01/2020

The Price of Tailoring the Index to Your Data: Poisoning Attacks on Learned Index Structures

The concept of learned index structures relies on the idea that the inpu...

Please sign up or login with your details

Forgot password? Click here to reset