DeepAI AI Chat
Log In Sign Up

GAAF: Searching Activation Functions for Binary Neural Networks through Genetic Algorithm

by   Yanfei Li, et al.

Binary neural networks (BNNs) show promising utilization in cost and power-restricted domains such as edge devices and mobile systems. This is due to its significantly less computation and storage demand, but at the cost of degraded performance. To close the accuracy gap, in this paper we propose to add a complementary activation function (AF) ahead of the sign based binarization, and rely on the genetic algorithm (GA) to automatically search for the ideal AFs. These AFs can help extract extra information from the input data in the forward pass, while allowing improved gradient approximation in the backward pass. Fifteen novel AFs are identified through our GA-based search, while most of them show improved performance (up to 2.54 testing on different datasets and network models. Our method offers a novel approach for designing general and application-specific BNN architecture. Our code is available at


page 1

page 2

page 3

page 4


Genetic Network Architecture Search

We propose a method for learning the neural network architecture that ba...

BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations

Binary Neural Networks (BNNs) have been garnering interest thanks to the...

Performance Analysis of Estimation of Distribution Algorithm and Genetic Algorithm in Zone Routing Protocol

In this paper, Estimation of Distribution Algorithm (EDA) is used for Zo...

Efficient Activation Function Optimization through Surrogate Modeling

Carefully designed activation functions can improve the performance of n...

Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distribution

Binarization of neural network models is considered as one of the promis...

Multi-objective Pruning for CNNs using Genetic Algorithm

In this work, we propose a heuristic genetic algorithm (GA) for pruning ...

High-performance Evolutionary Algorithms for Online Neuron Control

Recently, optimization has become an emerging tool for neuroscientists t...