Approximation of functions with one-bit neural networks

12/16/2021
by   C. Sinan Güntürk, et al.
0

This paper examines the approximation capabilities of coarsely quantized neural networks – those whose parameters are selected from a small set of allowable values. We show that any smooth multivariate function can be arbitrarily well approximated by an appropriate coarsely quantized neural network and provide a quantitative approximation rate. For the quadratic activation, this can be done with only a one-bit alphabet; for the ReLU activation, we use a three-bit alphabet. The main theorems rely on important properties of Bernstein polynomials. We prove new results on approximation of functions with Bernstein polynomials, noise-shaping quantization on the Bernstein basis, and implementation of the Bernstein polynomials by coarsely quantized neural networks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/10/2018

On the Universal Approximability of Quantized ReLU Neural Networks

Compression is a key step to deploy large neural networks on resource-co...
research
05/24/2019

Additive Noise Annealing and Approximation Properties of Quantized Neural Networks

We present a theoretical and experimental investigation of the quantizat...
research
09/18/2020

Searching for Low-Bit Weights in Quantized Neural Networks

Quantized neural networks with low-bit weights and activations are attra...
research
01/11/2022

Deep Neural Network Approximation For Hölder Functions

In this work, we explore the approximation capability of deep Rectified ...
research
02/01/2022

Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction

Memory footprint is one of the main limiting factors for large neural ne...
research
08/17/2022

Shallow neural network representation of polynomials

We show that d-variate polynomials of degree R can be represented on [0,...
research
04/05/2020

On Sharpness of Error Bounds for Multivariate Neural Network Approximation

Sharpness of error bounds for best non-linear multivariate approximation...

Please sign up or login with your details

Forgot password? Click here to reset