DeepAI AI Chat
Log In Sign Up

Matrix and tensor decompositions for training binary neural networks

by   Adrian Bulat, et al.

This paper is on improving the training of binary neural networks in which both activations and weights are binary. While prior methods for neural network binarization binarize each filter independently, we propose to instead parametrize the weight tensor of each layer using matrix or tensor decomposition. The binarization process is then performed using this latent parametrization, via a quantization function (e.g. sign function) applied to the reconstructed weights. A key feature of our method is that while the reconstruction is binarized, the computation in the latent factorized space is done in the real domain. This has several advantages: (i) the latent factorization enforces a coupling of the filters before binarization, which significantly improves the accuracy of the trained models. (ii) while at training time, the binary weights of each convolutional layer are parametrized using real-valued matrix or tensor decomposition, during inference we simply use the reconstructed (binary) weights. As a result, our method does not sacrifice any advantage of binary networks in terms of model compression and speeding-up inference. As a further contribution, instead of computing the binary weight scaling factors analytically, as in prior work, we propose to learn them discriminatively via back-propagation. Finally, we show that our approach significantly outperforms existing methods when tested on the challenging tasks of (a) human pose estimation (more than 4 (b) ImageNet classification (up to 5


XNOR-Net++: Improved Binary Neural Networks

This paper proposes an improved training algorithm for binary neural net...

Defensive Tensorization

We propose defensive tensorization, an adversarial defence technique tha...

Self-Distribution Binary Neural Networks

In this work, we study the binary neural networks (BNNs) of which both t...

Improving Binary Neural Networks through Fully Utilizing Latent Weights

Binary Neural Networks (BNNs) rely on a real-valued auxiliary variable W...

Learning from Non-Binary Constituency Trees via Tensor Decomposition

Processing sentence constituency trees in binarised form is a common and...


Neural networks have historically been built layerwise from the set of f...

A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks

The optimization of Binary Neural Networks (BNNs) relies on approximatin...