DeepAI AI Chat
Log In Sign Up

Estimating the effective dimension of large biological datasets using Fisher separability analysis

by   Luca Albergante, et al.
Musée Curie
Center for Research and Interdisciplinarity (CRI)

Modern large-scale datasets are frequently said to be high-dimensional. However, their data point clouds frequently possess structures, significantly decreasing their intrinsic dimensionality (ID) due to the presence of clusters, points being located close to low-dimensional varieties or fine-grained lumping. We test a recently introduced dimensionality estimator, based on analysing the separability properties of data points, on several benchmarks and real biological datasets. We show that the introduced measure of ID has performance competitive with state-of-the-art measures, being efficient across a wide range of dimensions and performing better in the case of noisy samples. Moreover, it allows estimating the intrinsic dimension in situations where the intrinsic manifold assumption is not valid.


Local intrinsic dimensionality estimators based on concentration of measure

Intrinsic dimensionality (ID) is one of the most fundamental characteris...

Estimating the intrinsic dimension of datasets by a minimal neighborhood information

Analyzing large volumes of high-dimensional data is an issue of fundamen...

Intrinsic dimension estimation for discrete metrics

Real world-datasets characterized by discrete features are ubiquitous: f...

Intrinsic Dimension for Large-Scale Geometric Learning

The concept of dimension is essential to grasp the complexity of data. A...

The role of intrinsic dimension in high-resolution player tracking data – Insights in basketball

A new range of statistical analysis has emerged in sports after the intr...

Observing the Population Dynamics in GE by means of the Intrinsic Dimension

We explore the use of Intrinsic Dimension (ID) for gaining insights in h...

Clustering by the local intrinsic dimension: the hidden structure of real-world data

It is well known that a small number of variables is often sufficient to...