Resilient Binary Neural Network

by   Sheng Xu, et al.

Binary neural networks (BNNs) have received ever-increasing popularity for their great capability of reducing storage burden as well as quickening inference time. However, there is a severe performance drop compared with real-valued networks, due to its intrinsic frequent weight oscillation during training. In this paper, we introduce a Resilient Binary Neural Network (ReBNN) to mitigate the frequent oscillation for better BNNs' training. We identify that the weight oscillation mainly stems from the non-parametric scaling factor. To address this issue, we propose to parameterize the scaling factor and introduce a weighted reconstruction loss to build an adaptive training objective. For the first time, we show that the weight oscillation is controlled by the balanced parameter attached to the reconstruction loss, which provides a theoretical foundation to parameterize it in back propagation. Based on this, we learn our ReBNN by calculating the balanced parameter based on its maximum magnitude, which can effectively mitigate the weight oscillation with a resilient training process. Extensive experiments are conducted upon various network models, such as ResNet and Faster-RCNN for computer vision, as well as BERT for natural language processing. The results demonstrate the overwhelming performance of our ReBNN over prior arts. For example, our ReBNN achieves 66.9 Top-1 accuracy with ResNet-18 backbone on the ImageNet dataset, surpassing existing state-of-the-arts by a significant margin. Our code is open-sourced at


page 1

page 2

page 3

page 4


Recurrent Bilinear Optimization for Binary Neural Networks

Binary Neural Networks (BNNs) show great promise for real-world embedded...

Training Binary Neural Networks with Real-to-Binary Convolutions

This paper shows how to train binary networks to within a few percent po...

BCNN: Binary Complex Neural Network

Binarized neural networks, or BNNs, show great promise in edge-side appl...

Rotated Binary Neural Network

Binary Neural Network (BNN) shows its predominance in reducing the compl...

Improving Binary Neural Networks through Fully Utilizing Latent Weights

Binary Neural Networks (BNNs) rely on a real-valued auxiliary variable W...

How Do Adam and Training Strategies Help BNNs Optimization?

The best performing Binary Neural Networks (BNNs) are usually attained u...

SiMaN: Sign-to-Magnitude Network Binarization

Binary neural networks (BNNs) have attracted broad research interest due...

Please sign up or login with your details

Forgot password? Click here to reset