DeepAI AI Chat
Log In Sign Up

BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations

by   Hyungjun Kim, et al.

Binary Neural Networks (BNNs) have been garnering interest thanks to their compute cost reduction and memory savings. However, BNNs suffer from performance degradation mainly due to the gradient mismatch caused by binarizing activations. Previous works tried to address the gradient mismatch problem by reducing the discrepancy between activation functions used at forward pass and its differentiable approximation used at backward pass, which is an indirect measure. In this work, we use the gradient of smoothed loss function to better estimate the gradient mismatch in quantized neural network. Analysis using the gradient mismatch estimator indicates that using higher precision for activation is more effective than modifying the differentiable approximation of activation function. Based on the observation, we propose a new training scheme for binary activation networks called BinaryDuo in which two binary activations are coupled into a ternary activation during training. Experimental results show that BinaryDuo outperforms state-of-the-art BNNs on various benchmarks with the same amount of parameters and computing cost.


page 12

page 14

page 17

page 18


Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distribution

Binarization of neural network models is considered as one of the promis...

Distance-aware Quantization

We address the problem of network quantization, that is, reducing bit-wi...

INSTA-BNN: Binary Neural Network with INSTAnce-aware Threshold

Binary Neural Networks (BNNs) have emerged as a promising solution for r...

Learning Aggregations of Binary Activated Neural Networks with Probabilities over Representations

Considering a probability distribution over parameters is known as an ef...

GAAF: Searching Activation Functions for Binary Neural Networks through Genetic Algorithm

Binary neural networks (BNNs) show promising utilization in cost and pow...

QuantNet: Learning to Quantize by Learning within Fully Differentiable Framework

Despite the achievements of recent binarization methods on reducing the ...

PokeBNN: A Binary Pursuit of Lightweight Accuracy

Top-1 ImageNet optimization promotes enormous networks that may be impra...