DeepAI AI Chat
Log In Sign Up

The Union of Manifolds Hypothesis and its Implications for Deep Generative Modelling

by   Bradley C. A. Brown, et al.
University of Waterloo
Layer 6 AI

Deep learning has had tremendous success at learning low-dimensional representations of high-dimensional data. This success would be impossible if there was no hidden low-dimensional structure in data of interest; this existence is posited by the manifold hypothesis, which states that the data lies on an unknown manifold of low intrinsic dimension. In this paper, we argue that this hypothesis does not properly capture the low-dimensional structure typically present in data. Assuming the data lies on a single manifold implies intrinsic dimension is identical across the entire data space, and does not allow for subregions of this space to have a different number of factors of variation. To address this deficiency, we put forth the union of manifolds hypothesis, which accommodates the existence of non-constant intrinsic dimensions. We empirically verify this hypothesis on commonly-used image datasets, finding that indeed, intrinsic dimension should be allowed to vary. We also show that classes with higher intrinsic dimensions are harder to classify, and how this insight can be used to improve classification accuracy. We then turn our attention to the impact of this hypothesis in the context of deep generative models (DGMs). Most current DGMs struggle to model datasets with several connected components and/or varying intrinsic dimensions. To tackle these shortcomings, we propose clustered DGMs, where we first cluster the data and then train a DGM on each cluster. We show that clustered DGMs can model multiple connected components with different intrinsic dimensions, and empirically outperform their non-clustered counterparts without increasing computational requirements.


page 18

page 19

page 20

page 22

page 23

page 24

page 26

page 27


On Deep Generative Models for Approximation and Estimation of Distributions on Manifolds

Generative networks have experienced great empirical successes in distri...

Intrinsic Dimension Estimation

It has long been thought that high-dimensional data encountered in many ...

Normalizing Flows Across Dimensions

Real-world data with underlying structure, such as pictures of faces, ar...

The Intrinsic Manifolds of Radiological Images and their Role in Deep Learning

The manifold hypothesis is a core mechanism behind the success of deep l...

Hierarchical Models: Intrinsic Separability in High Dimensions

It has long been noticed that high dimension data exhibits strange patte...

Manifold Topology Divergence: a Framework for Comparing Data Manifolds

We develop a framework for comparing data manifolds, aimed, in particula...

Estimating Distributions with Low-dimensional Structures Using Mixtures of Generative Models

There has been a growing interest in statistical inference from data sat...