Sample Complexity versus Depth: An Information Theoretic Analysis

03/01/2022
by   Hong Jun Jeon, et al.
0

Deep learning has proven effective across a range of data sets. In light of this, a natural inquiry is: "for what data generating processes can deep learning succeed?" In this work, we study the sample complexity of learning multilayer data generating processes of a sort for which deep neural networks seem to be suited. We develop general and elegant information-theoretic tools that accommodate analysis of any data generating process – shallow or deep, parametric or nonparametric, noiseless or noisy. We then use these tools to characterize the dependence of sample complexity on the depth of multilayer processes. Our results indicate roughly linear dependence on depth. This is in contrast to previous results that suggest exponential or high-order polynomial dependence.

READ FULL TEXT

page 1

page 2

page 3

page 4

12/18/2017

Size-Independent Sample Complexity of Neural Networks

We study the sample complexity of learning neural networks, by providing...
07/01/2020

Information Theoretic Sample Complexity Lower Bound for Feed-Forward Fully-Connected Deep Networks

In this paper, we study the sample complexity lower bound of a d-layer f...
11/22/2021

Private and polynomial time algorithms for learning Gaussians and beyond

We present a fairly general framework for reducing (ε, δ) differentially...
07/18/2022

On the Study of Sample Complexity for Polynomial Neural Networks

As a general type of machine learning approach, artificial neural networ...
10/02/2020

Hyperharmonic analysis for the study of high-order information-theoretic signals

Network representations often cannot fully account for the structural ri...
03/04/2018

Detecting Correlations with Little Memory and Communication

We study the problem of identifying correlations in multivariate data, u...
12/29/2018

Dependence Control at Large

We study the dependence control theory, with a focus on the tail propert...