DeepAI AI Chat
Log In Sign Up

Clustering by the local intrinsic dimension: the hidden structure of real-world data

by   Michele Allegra, et al.

It is well known that a small number of variables is often sufficient to effectively describe high-dimensional data. This number is called the intrinsic dimension (ID) of the data. What is not so commonly known is that the ID can vary within the same dataset. This fact has been highlighted in technical discussions, but seldom exploited to gain practical insight in the data structure. Here we develop a simple and robust approach to cluster regions with the same local ID in a given data landscape. Surprisingly, we find that many real-world data sets contain regions with widely heterogeneous dimensions. These regions host points differing in core properties: folded vs unfolded configurations in a protein molecular dynamics trajectory, active vs non-active regions in brain imaging data, and firms with different financial risk in company balance sheets. Our results show that a simple topological feature, the local ID, is sufficient to uncover a rich structure in high-dimensional data landscapes.


Estimating the intrinsic dimension of datasets by a minimal neighborhood information

Analyzing large volumes of high-dimensional data is an issue of fundamen...

Observing the Population Dynamics in GE by means of the Intrinsic Dimension

We explore the use of Intrinsic Dimension (ID) for gaining insights in h...

The role of intrinsic dimension in high-resolution player tracking data – Insights in basketball

A new range of statistical analysis has emerged in sports after the intr...

Intrinsic dimension estimation for locally undersampled data

High-dimensional data are ubiquitous in contemporary science and finding...

Estimating the effective dimension of large biological datasets using Fisher separability analysis

Modern large-scale datasets are frequently said to be high-dimensional. ...

Local intrinsic dimensionality estimators based on concentration of measure

Intrinsic dimensionality (ID) is one of the most fundamental characteris...

Hierarchical Models: Intrinsic Separability in High Dimensions

It has long been noticed that high dimension data exhibits strange patte...