. They demonstrated that deep convolutional neural networks trained end to end on large labelled datasets can beat most other techniques for image recognition. Deep learning quickly became the default algorithm for image classification and now even achieves super-human level performance
. Deep learning has also revolutionized other fields like speech recognition and natural language processing.
The two key ingredients needed to successfully apply deep neural networks are large amounts of labelled training data and powerful computing systems such as GPUs. Mobile devices including smartphones, Internet-of-Things (IoT) devices or smart home assistants have very limited processing power because of their intrinsic limitations on size and energy consumption. One possible solution is to offload all computations to the cloud but this introduces a latency and potentially even a privacy risk when sensitive data is processed remotely.
There is a considerable amount of active research on techniques to reduce the computational cost of deep learning models. One approach is to prune the network by removing redundant weights. The idea of pruning already goes back to the eighties when LeCun et al. used second order derivatives to calculate the impact of each weight on the loss of the network . Weights with a small impact are then removed from the network. More recently pruning was used on modern deep neural networks. Song et al. proposed a pruning pipeline where first weights with a small magnitude where removed after which the network was fine-tuned to recover the lost accuracy . Network pruning is especially effective when a network is used for transfer learning. In transfer learning the model is first trained on a large dataset such as Imagenet and is then fine-tuned on a small domain specific dataset.
Because the network was pretrained on a general dataset it will contain convolutional kernels that are not useful for the domain specific dataset. Molchanov et al. introduced a criterion based on a first-order Taylor expansion to decide which feature maps to remove and demonstrated impressive results when used together with transfer learning .
Instead of compressing a trained model it is also possible to train efficient models from scratch. The recently introduced MobileNets  use depthwise separable convolutions to reduce the computational cost. Depthwise separable convolutions factorize a standard convolution into a depthwise convolution and a 1x1 pointwise convolution. The depthwise convolution applies a single convolution to each input channel while the pointwise convolution combines the information in the different channels. Factorizing a traditional convolution into these two convolutions dramatically reduces the computational cost and size of the network with only a minimal reduction in accuracy.
Most implementations of deep neural networks use 32 bit floating point numbers for weights and activations. Various works have shown that this is not necessary and that it is possible to use 16 bit  or 8 bit  numbers. In the extreme case it is even possible to use binary weights and activations. Courbariaux et al. successfully trained convolutional neural networks for image recognition with binary weights and activations 
. This works surprisingly well for small scale datasets such as CIFAR10 but there is still a large drop in accuracy for larger datasets such as Imagenet. Neural networks with binary weights and activations are attractive because they replace the costly floating point multiplications and additions with bitwise XNORs and left and right bit shifts. These operations are very efficient to implement in hardware.
Another problem with deep learning is the need for large labelled datasets. Training a new model from scratch requires a large amount of training data. A well known technique is to use transfer learning where a model is first trained on a large dataset like Imagenet and afterwards the last layer is removed and retrained using a small amount of new domain specific training data. Transfer learning works because the first layers in the network learn to detect features such as color transitions and basic shapes that are present in images from different domains .
2 A hybrid hardware-software architecture
We propose a hybrid hardware-software architecture based on this idea of transfer learning. We train a neural network with binary weights and activations on the Imagenet dataset and use this network as a fixed feature extractor that could be optimised on hardware level. The last layer is implemented in software and is evaluated on the CPU (or GPU) of the device. Most of the computations are offloaded to the custom circuit. Since we only need to retrain the last layer of the network it even becomes feasible to train on the device itself instead of offloading the training to the cloud. This is very attractive from a privacy point of view because the training data never leaves the mobile device. Our architecture also allows app developers to embed custom neural networks into their apps. Right now this is often impossible because a typical neural network quickly requires hundreds of megabytes of storage just for the weights. If the device however is equipped with the fixed feature extractor we only need to ship the last (domain specific) layer of the network with the app. A schematic overview of our architecture is shown in Figure 1.
In this section we show that it is possible to use a binary neural network trained on Imagenet as a feature extractor for other datasets. We trained a binary version of the Alexnet architecture 
on the ILSVRC2012 training set following the binarization technique of Courbariaux et al.. Our binary network obtains a top 5 accuracy of 67% (42% top 1) while the floating point Alexnet obtains a top 5 accuracy of 80% (57% top 1). We then retrained the last layer on the domain specific datasets without changing the weights of the other layers. We report results for the three fine-grained image datasets summarized in Table 1
. For all our models we resized the input images such that the longest side was of length 256. During training we took random 224 by 224 pixel crops. For the test set we used center crops. The floating point networks where trained with stochastic gradient descent with momentum. For the binary networks we found that Adam gives better results which is consistent with Courbariaux et al. .
|Dataset||Number of classes||Training images||Testing images|
|UCSD Birds ||200||5,994||5,794|
|MIT Indoor scenes ||67||5.360||1,340|
The results are summarized in Table 2. The first part of the table shows the accuracies for the baseline floating point networks, either trained from scratch on the domain specific datasets or fine-tuned from the Imagenet model. As expected the fine-tuned models consistently outperform the models trained from scratch. The second part of the table shows the results for our binary models. The fine-tuned models again outperform the models trained from scratch which shows that transfer learning also works when the network uses binary weights and activations. For the last layer we can use binary weights and activations (a) but because this layer is evaluated in software in our proposed architecture we also experimented with floating point weights and activations (b). This consistently increases the accuracy on all datasets and pushes the accuracy closer to the accuracy of the floating point models.
|Top 1||Top 5||Top 1||Top 5||Top 1||Top 5|
|FP||trained from scratch||71.7%||90.5%||32.4%||57.2%||29.7%||56.4%|
|Binary||trained from scratch||63.2%||85.1%||13.8%||32.4%||29.0%||54.9%|
|(a) fine-tuned binary last layer||80.6%||94.5%||33.9%||62.9%||46.7%||77.4%|
|(b) fine-tuned FP last layer||84.0%||95.5%||36.7%||63.4%||48.3%||78.6%|
4 Conclusion and future work
We introduced a hybrid hardware-software architecture where a binary neural network trained on Imagenet can be embedded in a dedicated circuit. The last layer is implemented in software and is retrained for each specific task. We showed that transfer learning works very well for binary neural networks and experimented with a hybrid binary-floating point network where only the last layer uses floating point operations. This is a good trade-off between accuracy and computational cost since most of the computations can be offloaded to the fixed hardware accelerator. In future work we will explore other hybrid architectures. We now only considered finetuning the last layer of the network but we can also retrain more than one layer which should give a higher accuracy but will also incur a higher computational cost since a larger part of the network is evaluated in software.
We gratefully acknowledge the support of NVIDIA Corporation with the donation of GPU hardware used for this research. Steven Bohez is funded by a Ph.D. grant of the Agency for Innovation by Science and Technology in Flanders (IWT).
-  Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.
-  Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. CoRR, abs/1502.01852, 2015.
-  Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444, 2015.
-  Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In D. S. Touretzky, editor, Advances in Neural Information Processing Systems 2, pages 598–605. Morgan-Kaufmann, 1990.
-  Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pages 1135–1143, 2015.
-  Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient transfer learning. arXiv preprint arXiv:1611.06440, 2016.
-  Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
-  Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017.
-  Vincent Vanhoucke, Andrew Senior, and Mark Z Mao. Improving the speed of neural networks on cpus. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop, volume 1, page 4, 2011.
-  Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016.
-  Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pages 3320–3328, 2014.
-  Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
M-E. Nilsback and A. Zisserman.
Automated flower classification over a large number of classes.
Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
-  P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
Ariadna Quattoni and Antonio Torralba.
Recognizing indoor scenes.
Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 413–420. IEEE, 2009.