Clustered Hierarchical Entropy-Scaling Search of Astronomical and Biological Data
share
Both astronomy and biology are experiencing explosive growth of data, resulting in a "big data" problem standing in the way of a "big data" opportunity for discovery. One common task on such data sets is the problem of approximate search, or ρ-nearest neighbors search. We present a hierarchical search algorithm for such data sets that takes advantage of particular geometric properties apparent in both astronomical and biological data sets, namely the metric entropy and fractal dimensionality of the data. We present CHESS (Clustered Hierarchical Entropy-Scaling Search), a GPU-accelerated search tool with no loss in specificity or sensitivity, demonstrating a 6.4× speedup over linear search on the Sloan Digital Sky Survey's APOGEE data set and a 3.97× speedup on the GreenGenes 16S metagenomic data set, as well as asymptotically fewer comparisons on APOGEE when compared to the FALCONN locality-sensitive hashing library. CHESS allows for implicit data compression, which we demonstrate on the APOGEE data set. We also discuss an extension allowing for efficient k-nearest neighbors search.
READ FULL TEXT