2-D Embedding of Large and High-dimensional Data with Minimal Memory and Computational Time Requirements

by   Witold Dzwinel, et al.

In the advent of big data era, interactive visualization of large data sets consisting of M*10^5+ high-dimensional feature vectors of length N (N 10^3+), is an indispensable tool for data exploratory analysis. The state-of-the-art data embedding (DE) methods of N-D data into 2-D (3-D) visually perceptible space (e.g., based on t-SNE concept) are too demanding computationally to be efficiently employed for interactive data analytics of large and high-dimensional datasets. Herein we present a simple method, ivhd (interactive visualization of high-dimensional data tool), which radically outperforms the modern data-embedding algorithms in both computational and memory loads, while retaining high quality of N-D data embedding in 2-D (3-D). We show that DE problem is equivalent to the nearest neighbor nn-graph visualization, where only indices of a few nearest neighbors of each data sample has to be known, and binary distance between data samples -- 0 to the nearest and 1 to the other samples -- is defined. These improvements reduce the time-complexity and memory load from O(M log M) to O(M), and ensure minimal O(M) proportionality coefficient as well. We demonstrate high efficiency, quality and robustness of ivhd on popular benchmark datasets such as MNIST, 20NG, NORB and RCV1.


page 9

page 10


In search of the most efficient and memory-saving visualization of high dimensional data

Interactive exploration of large, multidimensional datasets plays a very...

The Future of Search and Discovery in Big Data Analytics: Ultrametric Information Spaces

Consider observation data, comprised of n observation vectors with value...

Visualization of Very Large High-Dimensional Data Sets as Minimum Spanning Trees

Here, we introduce a new data visualization and exploration method, TMAP...

Approximate kNN Classification for Biomedical Data

We are in the era where the Big Data analytics has changed the way of in...

Viewpoints: A high-performance high-dimensional exploratory data analysis tool

Scientific data sets continue to increase in both size and complexity. I...

Manifold learning with approximate nearest neighbors

Manifold learning algorithms are valuable tools for the analysis of high...

Please sign up or login with your details

Forgot password? Click here to reset