Sample Complexity versus Depth: An Information Theoretic Analysis

by   Hong Jun Jeon, et al.

Deep learning has proven effective across a range of data sets. In light of this, a natural inquiry is: "for what data generating processes can deep learning succeed?" In this work, we study the sample complexity of learning multilayer data generating processes of a sort for which deep neural networks seem to be suited. We develop general and elegant information-theoretic tools that accommodate analysis of any data generating process – shallow or deep, parametric or nonparametric, noiseless or noisy. We then use these tools to characterize the dependence of sample complexity on the depth of multilayer processes. Our results indicate roughly linear dependence on depth. This is in contrast to previous results that suggest exponential or high-order polynomial dependence.


page 1

page 2

page 3

page 4


Size-Independent Sample Complexity of Neural Networks

We study the sample complexity of learning neural networks, by providing...

Information Theoretic Sample Complexity Lower Bound for Feed-Forward Fully-Connected Deep Networks

In this paper, we study the sample complexity lower bound of a d-layer f...

Private and polynomial time algorithms for learning Gaussians and beyond

We present a fairly general framework for reducing (ε, δ) differentially...

On the Study of Sample Complexity for Polynomial Neural Networks

As a general type of machine learning approach, artificial neural networ...

Hyperharmonic analysis for the study of high-order information-theoretic signals

Network representations often cannot fully account for the structural ri...

Detecting Correlations with Little Memory and Communication

We study the problem of identifying correlations in multivariate data, u...

Dependence Control at Large

We study the dependence control theory, with a focus on the tail propert...