DeepAI AI Chat
Log In Sign Up

Defeating duplicates: A re-design of the LearnedSort algorithm

by   Ani Kristo, et al.
Brown University

LearnedSort is a novel sorting algorithm that, unlike traditional methods, uses fast ML models to boost the sorting speed. The models learn to estimate the input's distribution and arrange the keys in sorted order by predicting their empirical cumulative distribution function (eCDF) values. LearnedSort has shown outstanding performance compared to state-of-the-art sorting algorithms on several datasets, both synthetic and real. However, given the nature of the eCDF model, its performance is affected in the cases when the input data contains a large number of repeated keys (i.e., duplicates). This work analyzes this scenario in depth and introduces LearnedSort 2.0: a re-design of the algorithm that addresses this issue and enables the algorithm to maintain the leading edge even for high-duplicate inputs. Our extensive benchmarks on a large set of diverse datasets demonstrate that the new design performs at much higher sorting rates than the original version: an average of 4.78x improvement for high-duplicate datasets, and 1.60x for low-duplicate datasets while taking the lead among sorting algorithms.


page 6

page 8


Permutree sorting

Generalizing stack sorting and c-sorting for permutations, we define the...

Onesweep: A Faster Least Significant Digit Radix Sort for GPUs

We present Onesweep, a least-significant digit (LSD) radix sorting algor...

Apply Sorting Algorithms to FAST Problem

FAST problem is finding minimum feedback arc set problem in tournaments....

Engineering In-place (Shared-memory) Sorting Algorithms

We present sorting algorithms that represent the fastest known technique...

Lower Bounds for External Memory Integer Sorting via Network Coding

Sorting extremely large datasets is a frequently occuring task in practi...

Simple and Fast BlockQuicksort using Lomuto's Partitioning Scheme

This paper presents simple variants of the BlockQuicksort algorithm desc...

Sort-based grouping and aggregation

Database query processing requires algorithms for duplicate removal, gro...