A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off

06/03/2019 ∙ by Yaniv Blumenfeld, et al. ∙ 0

Reducing the precision of weights and activation functions in neural network training, with minimal impact on performance, is essential for the deployment of these models in resource-constrained environments. We apply mean-field techniques to networks with quantized activations in order to evaluate the degree to which quantization degrades signal propagation at initialization. We derive initialization schemes which maximize signal propagation in such networks and suggest why this is helpful for generalization. Building on these results, we obtain a closed form implicit equation for L_, the maximal trainable depth (and hence model capacity), given N, the number of quantization levels in the activation function. Solving this equation numerically, we obtain asymptotically: L_∝ N^1.82.



There are no comments yet.


page 8

page 18

page 19

page 20

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.