Depth Separation for Neural Networks

02/27/2017
by   Amit Daniely, et al.
0

Let f:S^d-1×S^d-1→S be a function of the form f(x,x') = g(〈x,x'〉) for g:[-1,1]→R. We give a simple proof that shows that poly-size depth two neural networks with (exponentially) bounded weights cannot approximate f whenever g cannot be approximated by a low degree polynomial. Moreover, for many g's, such as g(x)=sin(π d^3x), the number of neurons must be 2^Ω(d(d)). Furthermore, the result holds w.r.t. the uniform distribution on S^d-1×S^d-1. As many functions of the above form can be well approximated by poly-size depth three networks with poly-bounded weights, this establishes a separation between depth two and depth three networks w.r.t. the uniform distribution on S^d-1×S^d-1.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/31/2020

Neural Networks with Small Weights and Depth-Separation Barriers

In studying the expressiveness of neural networks, an important question...
research
04/15/2019

Depth Separations in Neural Networks: What is Actually Being Separated?

Existing depth separation results for constant-depth networks essentiall...
research
05/25/2023

Most Neural Networks Are Almost Learnable

We present a PTAS for learning random constant-depth networks. We show t...
research
01/07/2020

Poly-time universality and limitations of deep learning

The goal of this paper is to characterize function distributions that de...
research
08/05/2022

Towards Antisymmetric Neural Ansatz Separation

We study separations between two fundamental models (or Ansätze) of anti...
research
05/18/2018

Approximate Data Depth Revisited

Halfspace depth and β-skeleton depth are two types of depth functions in...
research
02/02/2021

Depth separation beyond radial functions

High-dimensional depth separation results for neural networks show that ...

Please sign up or login with your details

Forgot password? Click here to reset