
Signal propagation in continuous approximations of binary neural networks
The training of stochastic neural network models with binary (±1) weight...
read it

Where Should We Begin? A LowLevel Exploration of Weight Initialization Impact on Quantized Behaviour of Deep Neural Networks
With the proliferation of deep convolutional neural network (CNN) algori...
read it

Proximal Meanfield for Neural Network Quantization
Compressing large neural networks by quantizing the parameters, while ma...
read it

Deep Information Propagation
We study the behavior of untrained neural networks whose weights and bia...
read it

Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
Training recurrent neural networks (RNNs) on long sequence tasks is plag...
read it

Exact information propagation through fullyconnected feed forward neural networks
Neural network ensembles at initialisation give rise to the trainability...
read it

Towards Efficient Training for Neural Network Quantization
Quantization reduces computation costs of neural networks but suffers fr...
read it
A Mean Field Theory of Quantized Deep Networks: The QuantizationDepth TradeOff
Reducing the precision of weights and activation functions in neural network training, with minimal impact on performance, is essential for the deployment of these models in resourceconstrained environments. We apply meanfield techniques to networks with quantized activations in order to evaluate the degree to which quantization degrades signal propagation at initialization. We derive initialization schemes which maximize signal propagation in such networks and suggest why this is helpful for generalization. Building on these results, we obtain a closed form implicit equation for L_, the maximal trainable depth (and hence model capacity), given N, the number of quantization levels in the activation function. Solving this equation numerically, we obtain asymptotically: L_∝ N^1.82.
READ FULL TEXT
Comments
There are no comments yet.