Information Theoretic Sample Complexity Lower Bound for Feed-Forward Fully-Connected Deep Networks

07/01/2020
by   Xiaochen Yang, et al.
0

In this paper, we study the sample complexity lower bound of a d-layer feed-forward, fully-connected neural network for binary classification, using information-theoretic tools. Specifically, we propose a backward data generating process, where the input is generated based on the binary output, and the network is parametrized by weight parameters for the hidden layers. The sample complexity lower bound is of order Ω(log(r) + p / (r d)), where p is the dimension of the input, r is the rank of the weight matrices, and d is the number of hidden layers. To the best of our knowledge, our result is the first information theoretic sample complexity lower bound.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/07/2021

A Lower Bound for the Sample Complexity of Inverse Reinforcement Learning

Inverse reinforcement learning (IRL) is the task of finding a reward fun...
research
01/28/2016

Information-Theoretic Lower Bounds for Recovery of Diffusion Network Structures

We study the information-theoretic lower bound of the sample complexity ...
research
03/07/2019

Limiting Network Size within Finite Bounds for Optimization

Largest theoretical contribution to Neural Networks comes from VC Dimens...
research
06/29/2017

Comparing Information-Theoretic Measures of Complexity in Boltzmann Machines

In the past three decades, many theoretical measures of complexity have ...
research
04/18/2018

Understanding Individual Neuron Importance Using Information Theory

In this work, we characterize the outputs of individual neurons in a tra...
research
06/19/2022

Artificial intelligence system based on multi-value classification of fully connected neural network for construction management

This study is devoted to solving the problem to determine the profession...
research
03/01/2022

Sample Complexity versus Depth: An Information Theoretic Analysis

Deep learning has proven effective across a range of data sets. In light...

Please sign up or login with your details

Forgot password? Click here to reset