INSTA-BNN: Binary Neural Network with INSTAnce-aware Threshold

04/15/2022
by   Changhun Lee, et al.
0

Binary Neural Networks (BNNs) have emerged as a promising solution for reducing the memory footprint and compute costs of deep neural networks. BNNs, on the other hand, suffer from information loss because binary activations are limited to only two values, resulting in reduced accuracy. To improve the accuracy, previous studies have attempted to control the distribution of binary activation by manually shifting the threshold of the activation function or making the shift amount trainable. During the process, they usually depended on statistical information computed from a batch. We argue that using statistical data from a batch fails to capture the crucial information for each input instance in BNN computations, and the differences between statistical information computed from each instance need to be considered when determining the binary activation threshold of each instance. Based on the concept, we propose the Binary Neural Network with INSTAnce-aware threshold (INSTA-BNN), which decides the activation threshold value considering the difference between statistical data computed from a batch and each instance. The proposed INSTA-BNN outperforms the baseline by 2.5 classification task with comparable computing cost, achieving 68.0 top-1 accuracy on ResNet-18 and MobileNetV1 based models, respectively.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/02/2020

Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distribution

Binarization of neural network models is considered as one of the promis...
research
02/16/2020

BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations

Binary Neural Networks (BNNs) have been garnering interest thanks to the...
research
02/02/2019

Self-Binarizing Networks

We present a method to train self-binarizing neural networks, that is, n...
research
03/23/2019

BitSplit-Net: Multi-bit Deep Neural Network with Bitwise Activation Function

Significant computational cost and memory requirements for deep neural n...
research
05/24/2019

Dichotomize and Generalize: PAC-Bayesian Binary Activated Deep Neural Networks

We present a comprehensive study of multilayer neural networks with bina...
research
10/06/2022

IR2Net: Information Restriction and Information Recovery for Accurate Binary Neural Networks

Weight and activation binarization can efficiently compress deep neural ...
research
11/30/2021

PokeBNN: A Binary Pursuit of Lightweight Accuracy

Top-1 ImageNet optimization promotes enormous networks that may be impra...

Please sign up or login with your details

Forgot password? Click here to reset