NetScore: Towards Universal Metrics for Large-scale Performance Analysis of Deep Neural Networks for Practical Usage

06/14/2018 ∙ by Alexander Wong, et al. ∙ 0

Much of the focus in the design of deep neural networks has been on improving accuracy, leading to more powerful yet highly complex network architectures that are difficult to deploy in practical scenarios, particularly on edge devices such as mobile and other consumer devices, given their high computational and memory requirements. As a result, there has been a recent interest in the design of quantitative metrics for evaluating deep neural networks that accounts for more than just model accuracy as the sole indicator of network performance. In this study, we continue the conversation towards universal metrics for evaluating the performance of deep neural networks for practical usage. In particular, we propose a new balanced metric called NetScore, which is designed specifically to provide a quantitative assessment of the balance between accuracy, computational complexity, and network architecture complexity of a deep neural network. In what is one of the largest comparative analysis between deep neural networks in literature, the NetScore metric, the top-1 accuracy metric, and the popular information density metric were compared across a diverse set of 50 different deep convolutional neural networks for image classification on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC 2012) dataset. The evaluation results across these three metrics for this diverse set of networks are presented in this study to act as a reference guide for practitioners in the field. The proposed NetScore metric, along with the other tested metrics, are by no means perfect, but the hope is to push the conversation towards better universal metrics for evaluating deep neural networks for use in practical scenarios to help guide practitioners in model design.

READ FULL TEXT VIEW PDF

Authors

page 5

page 8

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

There has been a recent urge in both research and industrial interests in deep learning 

lecun2015deep , with deep neural networks demonstrating state-of-the-art performance in recent years across a wide variety of applications. In particular, deep convolutional neural networks lecun1 ; lecun2

has been shown to outperform other machine learning approaches for visual perception tasks ranging from image classification 

krizhevsky2012imagenet to object detection liu2016ssd and segmentation MaskRCNN . One of the key driving factors behind the tremendous recent successes in deep neural networks has been the availability of massive computing resources thanks to the advances and proliferation of cloud computing and highly parallel computing hardware such as graphics processing units (GPUs). The availability of this wealth of computing resources has enabled researchers to explore significantly more complex and increasingly deeper neural networks that has resulted in significant performance gains over past machine learning methods. For example, in the realm of visual perception, the depth of deep convolutional neural networks with state-of-the-art accuracies have reached hundreds of layers, hundreds of millions of parameters in size, and billions of calculations for inferencing.

While the ability to build such large and complex deep neural networks has led to a constant increase in accuracy, the primary metric for performance widely leveraged for evaluating networks, it has also created significant barriers to the deployment of such networks for practical edge device usage. The practical deployment bottlenecks associated with the powerful yet highly complex deep neural networks in research literature has become even more visible in recent years due to the incredible proliferation of mobile devices, consumer devices, and other edge devices and the increasing demand for machine learning applications in such devices. As a result, the design of deep neural networks that account for more than just accuracy as the sole indicator of network performance and instead strike a strong balance between accuracy and complexity has very recently become a very hot area of research focus, with a number of different deep neural network architectures designed specifically with efficiency in mind iandola2016squeezenet ; howard2017mobilenets ; tinyssd ; munet ; sandler2018mobilenetv2 ; squishednet ; ShuffleNet .

One of the key challenges in designing deep neural networks that strikes a strong balance between accuracy and complexity for practical usage lies in the difficulties with assessing how well a particular network architecture is striking that balance. As previous mentioned, using accuracy as the sole metric for network performance does not provide the proper indicators of how efficient a particular network is in practical scenarios such as deployment on mobile devices and other consumer devices. As a result, there has been a recent interest in the design of quantitative metrics for evaluating deep neural networks that accounts for more than just model accuracy. In particular, it is generally desirable to design such metrics in a manner that is as hardware vendor agnostic as possible so that different network architectures can be compared to each other in a consistent manner. One of the most widely cited metrics in research literature for assessing the performance of deep neural networks that accounts for both accuracy and architectural complexity is the information density metric proposed by Canziani , which attempts to measure the relative amount of accuracy captured within one of the most basic building blocks of a deep neural network: a parameter. More specifically, the information density () of a deep neural network is defined as the accuracy of the deep neural network (denoted by ) divided by the number of parameters needed for representing it (denoted by ),

(1)

While highly effective for giving a good general idea of the balance between accuracy and architectural complexity (which also acts as a good indicator for memory requirements), the information density metric does not account for the fact that, depending on the design of the network architecture, the architecture complexity does not necessarily reflect the computational requirements for performing network inference (e.g., MobileNet howard2017mobilenets has more parameters than SqueezeNet iandola2016squeezenet but has lower computational requirements for network inference). Therefore, the exploration and investigation towards universal performance metrics that account for accuracy, architectural complexity, and computational complexity is highly desired as it has the potential to improve network model search and design.

In this study, we continue the conversation towards universal metrics for evaluating the performance of deep neural networks for practical usage. In particular, we propose a new balanced metric called NetScore, which is designed specifically to provide a quantitative assessment of the balance between accuracy, computational complexity, and network architecture complexity of a deep neural network. This paper is organized as follows. Section 2 describes the proposed NetScore metric and the design principles around it. Section 3 presents and discusses experimental results that compare the NetScore, information density, and top-1 accuracy across 60 different deep convolutional neural networks for image classification on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC 2012) dataset ImageNet , making this one of the largest comparative studies between deep neural networks.

2 NetScore: Design Principles

The proposed NetScore metric (denoted here as ) for assessing the performance of a deep neural network for practical usage can be defined as:

(2)

where is the accuracy of the network, is the number of parameters in the network, is the number of multiply–accumulate (MAC) operations performed during network inference, and , , are coefficients that control the influence of accuracy, architectural complexity, and computational complexity of the network on . A number of design principles were taken into consideration in the design of the proposed NetScore metric, which is described below.

Model accuracy representation: In the NetScore metric, the obvious incorporation of the model accuracy of the network into the metric is in the numerator of the ratio, as an increase in accuracy should naturally lead to an increase in the metric, similar to the information density metric Canziani . We further introduce a coefficient in the proposed NetScore metric to provide better control over the influence of model accuracy on the overall metric. In particular, we set to better emphasize the importance of model accuracy in assessing the overall performance of a network in practical usage, as deep convolutional neural networks that have unreasonably low model accuracy remain unusable in practical scenarios, regardless how small or fast the network is. In this study, the unit used for is in percent top-1 accuracy on the ILSVRC 2012 dataset ImageNet .

Model architectural and computational complexity representations: Taking inspiration from the information density metric Canziani , we represent the architectural complexity of a deep neural network by the number of parameters in the network and incorporate it in the denominator of the ratio. As such, the architecture complexity of the network is inversely proportional to the metric , where an increase in architectural complexity results in a decrease in . In addition, we incorporate the computational complexity of the deep neural network as an additional factor in the denominator of the ratio to be taken into consideration for assessing the overall performance of a network for practical usage, which is particularly important in operational scenarios such as inference on mobile devices and other consumer devices where computational power is limited. To represent the computational complexity of the network in a manner that is relatively hardware vendor agnostic, thus enabling a more consistent comparison between networks, we chose to leverage the number of multiply–accumulate (MAC) operations necessary for performing network inference. Given that the computational bottleneck associated with performing network inference on a deep neural network is predominantly in the computation of MAC operations, the number of MAC operations is a good proxy for the computational complexity of the network. By incorporating both architectural and computational complexity, the proposed NetScore metric can better quantify the balance between accuracy, memory requirements, and computational requirements in practical usage. Furthermore, we introduce two coefficients ( and , respectively) to provide better control over the influence of architectural and computational complexity on the overall metric. In particular, we set and since, while architectural and computational complexity are both very important factors to assessing the overall performance of a network in practical scenarios, the most important metric remains the model accuracy given that, as eluded to before, networks with unreasonably low model accuracy are not useful in practical scenarios regardless of size and speed.

Logarithmic scaling: One of the difficulties with comparing the overall performance of different deep neural networks with each other is their great diversity in their model accuracy, architectural complexity, and computational complexity. This makes the dynamic range of the performance metric quite large and unwieldy for practitioners to compare for model search and design purposes. To account for this large dynamic range, we take inspiration from the field of signal processing; in particular, the decibel scale commonly used to express the ratio between one value of a property to another on a logarithmic scale. In the proposed NetScore metric, we transform the ratio between the model accuracy property () and the model architectural and computational complexity ( and ) into the logarithmic decibel scale to reduce the dynamic range to within a more readily interpretable range.

3 Experimental Results and Discussion

To get a better sense regarding the overall performance of the huge wealth of deep convolutional neural networks introduced in research literature in the context of practical usage, we perform a large-scale comparative analysis across a diverse set of 60 different deep convolutional neural networks designed for image classification using the following quantitative performance metrics: i) top-1 accuracy, ii) information density, and iii) the proposed NetScore metric. The dataset of choice for the comparative analysis in this study is the ImageNet Large Scale Visual Recognition Challenge (ILSVRC 2012) dataset ImageNet , which consists of 1000 different classes. To the best of the author’s knowledge, this comparative analysis is one of the largest in research literature and the hope is that the results presented in this study can act as a reference guide for practitioners in the field.

The set of deep convolutional neural networks being evaluated in this study are: AlexNet krizhevsky2012imagenet , AmoebaNet-A (4, 50) AmoebaNet , AmoebaNet-A (6, 190) AmoebaNet , AmoebaNet-A (6, 204) AmoebaNet , AmoebaNet-B (3, 62) AmoebaNet , AmoebaNet-B (6, 190) AmoebaNet , AmoebaNet-C (4, 50) AmoebaNet , AmoebaNet-C (6, 228) AmoebaNet , CondenseNet (G=C=4) CondenseNet , CondenseNet (G=C=8) CondenseNet , DenseNet-121 (k=32) densenet , DenseNet-169 (k=32) densenet , DenseNet-161 (k=48) densenet , DenseNet-201 (k=32) densenet , DPN-131 DPN , GoogleNet GoogleNet , IGC-L100M2 IGC , IGC-L16M16 IGC , IGC-L100M2 IGC , Inception-ResNetv2 Inceptionv4 , Inceptionv2 Inceptionv2 , Inceptionv3 Inceptionv2 , Inceptionv4 Inceptionv4 , MobileNetv1 (1.0-224) howard2017mobilenets , MobileNetv1 (1.0-192) howard2017mobilenets , MobileNetv1 (1.0-160) howard2017mobilenets , MobileNetv1 (1.0-128) howard2017mobilenets , MobileNetv1 (0.75-224) howard2017mobilenets , MobileNetv2 sandler2018mobilenetv2 , MobileNetv2 (1.4) sandler2018mobilenetv2 , NASNet-A (4 @ 1056) NASNet , NASNet-A (6 @ 4132) NASNet , NASNet-B (4 @ 1536) NASNet , NiN NiN , OverFeat Overfeat , PNASNet-5 (4, 216) PNASNet , PolyNet PolyNet , PreResNet-152 preresnet , PreResNet-200 preresnet , PyramidNet-101 (alpha=250) PyramidNet , PyramidNet-200 (alpha=300) PyramidNet , PyramidNet-200 (alpha=450) PyramidNet , ResNet-152 ResNet , ResNet-50 ResNet , ResNet-101 ResNet , ResNeXt-101, SENet SENet , ShuffleNet (1.5) ShuffleNet , ShuffleNet (x2) ShuffleNet , SimpleNet SimpleNet , SqueezeNet iandola2016squeezenet , SqueezeNetv1.1 iandola2016squeezenet , SqueezeNext (1.0-23v5) SqueezeNext , SqueezeNext (2.0-23) SqueezeNext , SqueezeNext (2.0-23v5) SqueezeNext , TinyDarkNet TinyDarkNet , VGG16 VGG16 , Xception Xception , ZynqNet ZynqNet .

In this study, the units used for and for two of the quantitative performance metrics (information density and the proposed NetScore metric) are in M-Params (millions of parameters) and G-MACs (billions of MAC operations), respectively, given that most modern deep convolutional neural networks are within those architectural and computational complexity ranges.

Top-1 accuracy: The top-1 accuracies across 60 different deep convolutional neural networks for the ILSVRC 2012 dataset is shown in Fig. 1. It can be clearly observed that significant progress has been made in the design of deep convolutional neural networks for image classification over the past six years, with the difference between the deep convolutional neural network with the highest top-1 accuracy in this study (i.e., AmoebaNet-C (6, 228)) and that of AlexNet exceeding 25%. It is also interesting to see that more recent developments in efficient deep convolutional neural networks such as MobileNetv1, MobileNetv2, and ShuffleNet all have top-1 accuracies that exceed VGG-16, the third largest tested network evaluated in the study that was also the state-of-the-art just four years ago, thus further illustrating the improvements in network design over the past few years.

Figure 1: Top-1 accuracy across 60 different deep convolutional neural networks for the ILSVRC 2012 dataset.

Information density: The information densities across 60 different deep convolutional neural networks for the ILSVRC 2012 dataset is shown in Fig. 2. It can be clearly observed that the deep convolutional neural networks that were specifically designed for efficiency (e.g., MobileNetv1, MobileNetv2, ShuffleNet, SqueezeNet, Tiny DarkNet, and SqueezeNext) have significantly higher information densities compared to networks that were designed purely with accuracy as a metric. More specifically, the SqueezeNext (1.0-23v5), Tiny DarkNet, and the SqueezeNet family of networks had the highest information density by a wide margin compared to the other tested deep convolutional neural networks, which can be attributed to their significantly lower architectural complexity in terms of number of network parameters. Another notable observation from the results in Fig. 2 is that the dynamic range of the information density metric is quite large across the diverse set of 60 deep convolutional neural networks evaluated in this study.

Figure 2: Information density across 60 different deep convolutional neural networks for the ILSVRC 2012 dataset. Units are in %/M-Params.
Figure 3: NetScore across 60 different deep convolutional neural networks for the ILSVRC 2012 dataset.

NetScore: The NetScore across 60 different deep convolutional neural networks for the ILSVRC 2012 dataset is shown in Fig. 3. Similar to the trend observed in Fig. 2, it can be clearly observed that many of the deep convolutional neural networks that were specifically designed for efficiency have significantly higher NetScores compared to networks that were designed purely with accuracy as a metric. However, what is interesting to observe is that the NetScore ranking amongst these efficient networks are quite different than that when using the information density metric. In particular, the top ranking deep convolutional neural networks with the highest NetScores are SqueezeNext (1.0-23v5), CondenseNet (G=C=8), and MobileNetv2.

The SqueezeNet family of networks, on the other hand, had much lower relative NetScores compared to the aforementioned efficient networks despite having the top two highest information densities. This observation illustrates the effect of incorporating computational complexity to the assessment of deep convolutional neural networks for practical usage, given that while the SqueezeNet family of networks has significantly lower architectural complexities compared to other tested networks, it also is offset by noticeably higher computational complexities compared to other tested efficient networks such as the MobileNetv1, MobileNetv2, SqueezeNext, and ShuffleNet network families.

The proposed NetScore metric, which by no means is perfect, could potentially be useful for guiding practitioners in model search and design and hopefully push the conversation towards better universal metrics for evaluating deep neural networks for use in practical scenarios. Future work includes incorporating additional or alternative factors that are important to assessing architectural and computational complexities of deep neural networks beyond what is being used in the NetScore metric, as well as finding a good balance between these different factors based on relative importance for the deployment of deep neural networks for practical usage in scenarios such as mobile devices and other edge devices.

Acknowledgment

The author thank Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada Research Chairs program, Nvidia, and DarwinAI.

References