Competitions encourage diligent development of advanced technology. Historical examples include Ansari XPRIZE competitions for suborbital spaceflight, numerous Kaggle competitions such as identifying salt deposits beneath the Earth’s surface from seismic images, and the PASCAL VOC, ILSVRC, and COCO competitions for computer vision PASCAL ; ILSVRC15 ; COCO . The IEEE International Low-Power Image Recognition Challenge (LPIRC, https://rebootingcomputing.ieee.org/lpirc) accelerates the development of computer vision solutions that are low-latency, accurate, and low-power.
Started in , LPIRC is an annual competition identifying the best system-level solution for detecting objects in images while using as little energy as possible 7372672 ; 8342099 ; 7858303 ; arxiv . Although many competitions are held every year, LPIRC is the only one integrating both computer vision and low power. In LPIRC, a contestants’ system is connected to a referee system through an intranet (wired or wireless). In , the competition has three tracks. For track
, teams submit neural network architectures optimized by Google’s TfLite engine and executed on Google’s Pixelphone. For track
, teams submit neural network architectures coded in Caffeand executed on NVIDIA Jetson TX. For track , teams optimize both software and hardware and bring the end system on-site for evaluation. We highlight track 1 below for it has enlisted a large number of high-quality solutions.
2 Track 1: Efficient Network Architectures for Mobile
Track 1’s goal is to help contestants develop real-time image classification on high-end mobile phones. The platform simplifies the development cycles by providing an automated benchmarking service. Once a model is submitted in Tensorflow format, the service uses TfLite to optimize it for on-device deployment and then dispatches the model to a Pixel 2 device for latency and accuracy measurements. The service also ensures that all models are benchmarked in the same environment for reproducibility and comparability.
Track 1 selects submissions with the best accuracy within a ms-per-image time constraint, using a batch size of and a single big core in Pixel 2. Although no power or energy is explicitly measured, latency correlates reasonably with energy consumption. Table 1 shows the score of the track
winner’s solution. The model is evaluated on both the ILSVRC2012-ImageNetILSVRC15 validation set as well as a freshly collected holdout set.
|Average Latency||Test Metrics||Accuracy on Classified||Accuracy/Time||Number Classified|
Track received a total of valid submissions (submissions that passed the bazel test and successfully evaluated) and submissions received test metric scores between and . Slightly over half (%) of the solutions use -bit quantization. Most of the architectures (%) are variations of the existing Mobilenet model family, namely quantized V (%), quantized V (%) and float V (%). The winning track submission outperformed the previous state-of-the-art below ms (based on quantized MobileNet V) in accuracy by %. The predominant dependence on Mobilenets is expected considering their exceptional on-device performance and technical support, although future installments are looking to mechanistically discover novel architectures.
3 Winning Track 1 with Quantization-Friendly Mobilenets
3.1 Large Quantization Loss in Precision
The winning solution is based on MobileNet V, but modified in a way that is quantization-friendly. Quantization is often critical for low latency inference on mobile. As most neural networks are trained using floating-point models, they need to be converted to fixed-point in order to efficiently run on mobile devices. Although Google’s MobileNet models successfully reduce the parameter size and computational latency by using separable convolution, direct post-quantization on a pre-trained MobileNet V model can result in significant precision loss. For example, the accuracy of a quantized MobileNet V could drop to % on ImageNet validation data-set as shown in Table 2.
|Image Validation Accuracy|
|Model||Floating-point||Bit Fixed-point||Floating-point||Bit Fixed-point|
|(COCO)||(COCO )||(COCO )||(COCO )||(COCO )|
The root cause of accuracy loss due to quantization in such separable convolution networks is analyzed as follows. In separable convolutions, depth-wise convolution is applied on each channel independently, while the min and max values used for weights quantization are taken collectively from all channels. Furthermore, without correlation crossing channels, depth-wise convolution may be prone to produce all-zero values of weights in one channel. All-zero values in one channel have very small variance which leads to a large “scale” value for that specific channel when applying batch normalization transform directly after depth-wise convolution. Therefore, such outliers in one channel may cause a large quantization loss for the whole model due to uneven distributed data range. This is commonly observed in both MobileNet VQuantization and V models. Figure 1 shows an example of the observed batch normalization scale values of channels extracted from the first depth-wise convolution layer in MobileNetV float model. As a result, those small values corresponding to informative channels are not well preserved after quantization and this significantly reduces the representation power of the model.
3.2 The Winning Quantization-Friendly Approach
For a better solution, an effective quantization-friendly separable convolution architecture is proposed as shown in Figure 2
(c), where the non-linear operations (both batch normalization and ReLU) between depth-wise and point-wise convolution layers are all removed, letting the network learn proper weights to handle the batch normalization transform directly. In addition, ReLU is replaced with ReLU in all point-wise convolution layers. From the experiments in MobileNet V and V models, this architecture maintains high accuracy in the -bit quantized pipeline in various tasks such as image recognition and object detection.
As an alternative, one can use Learn2Compress learn2compress , Google’s ML framework for directly training efficient on-device models from scratch or an existing TensorFlow model by combining quantization along with other techniques like distillation, pruning, and joint training. Comparing with these options, the winners’ solution provides a much simpler way to modify separable convolution layers and make whole network quantization-friendly without re-training.
3.3 System Integration and Experimental Results
By considering the trade-off between accuracy and model complexity, MobileNetV__ is chosen as the base architecture to apply the quantization-friendly changes. Based on the proposed structure, a floating-point model can be trained on the dataset. During the post-quantization step, the model runs against a range of different inputs, one image in each class category from the training data, to collect min and max values as well as the data histogram distribution at each layer output. Values for optimal “step size” and “offset”, represented by and , that minimize the summation of quantization loss and saturation loss during a greedy search, are picked for linear quantization. Given the calculated range of min and max values, TensorFlow Lite provides a path to convert a graph model (.pb) to tflite model (.tflite) that can be deployed on edge devices.
The proposed fixed-point model with an input resolution of 128 can achieve an accuracy of % on ImageNet validation dataset and an accuracy of % on holdout dataset. The base network model can also be used for different tasks such as image recognition and object detection. Table 2 shows our experimental results on the accuracy of the proposed quantization-friendly MobileNets for image recognition on ImageNet (with an input resolution of ) and object detection on COCO dataset. With the proposed core network structure, the model largely increases accuracy on both tasks in bit fixed-point pipeline. Whereas direct quantization on pre-trained MobileNet V and V model would cause unacceptable accuracy loss, the quantization-friendly MobileNets managed to stay within of the float model’s accuracy for ImageNet classification, and within for COCO object detection.
By providing an convenient platform for evaluating on-device neural network architectures, LPIRC has successfully enlisted creative solutions from the field. Not only does the winning solution outperformed state-of-the-art, it also provides insights regarding quantization that is applicable to other tasks such as object detection. This success showcases not only the effectiveness of their quantization-friendly approach, but also the importance of the platform that facilitates its development.
- (1) Everingham et al. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98–136, jan 2015.
- (2) Russakovsky et al. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3), December 2015.
- (3) Lin et al. Microsoft COCO: Common objects in context. In ECCV, 2014.
- (4) Lu et al. Rebooting computing and low-power image recognition challenge. In IEEE/ACM International Conference on Computer-Aided Design, 2015.
- (5) Gauen et al. Three years of low-power image recognition challenge: Introduction to special session. In Design, Automation Test in Europe Conference Exhibition, 2018.
- (6) Gauen et al. Low-power image recognition challenge. In Asia and South Pacific Design Automation Conference, 2017.
- (7) Alyamkin et al. 2018 low-power image recognition challenge. In https://arxiv.org/abs/1810.01732, 2018.
- (8) Sheng et al. A quantization-friendly separable convolution for mobilenets. In https://arxiv.org/abs/1803.08607, March 2018.
- (9) Sujith Ravi. Custom on-device ml models with learn2compress.