Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoT

07/26/2019
by   Kartikeya Bhardwaj, et al.
4

Model compression has emerged as an important area of research for deploying deep learning models on Internet-of-Things (IoT). However, for extremely memory-constrained scenarios, even the compressed models cannot fit within the memory of a single device and, as a result, must be distributed across multiple devices. This leads to a distributed inference paradigm in which memory and communication costs represent a major bottleneck. Yet, existing model compression techniques are not communication-aware. Therefore, we propose Network of Neural Networks (NoNN), a new distributed IoT learning paradigm that compresses a large pretrained 'teacher' deep network into several disjoint and highly-compressed 'student' modules, without loss of accuracy. Moreover, we propose a network science-based knowledge partitioning algorithm for the teacher model, and then train individual students on the resulting disjoint partitions. Extensive experimentation on five image classification datasets, for user-defined memory/performance budgets, show that NoNN achieves higher accuracy than several baselines and similar accuracy as the teacher model, while using minimal communication among students. Finally, as a case study, we deploy the proposed model for CIFAR-10 dataset on edge devices and demonstrate significant improvements in memory footprint (up to 24x), performance (up to 12x), and energy per node (up to 14x) compared to the large teacher model. We further show that for distributed inference on multiple edge devices, our proposed NoNN model results in up to 33x reduction in total latency w.r.t. a state-of-the-art model compression baseline.

READ FULL TEXT

page 4

page 9

page 13

page 15

page 17

page 18

page 20

page 21

research
11/03/2021

Communication-Efficient Separable Neural Network for Distributed Inference on Edge Devices

The inference of Neural Networks is usually restricted by the resources ...
research
07/22/2023

MIMONet: Multi-Input Multi-Output On-Device Deep Learning

Future intelligent robots are expected to process multiple inputs simult...
research
06/12/2019

Visual Wake Words Dataset

The emergence of Internet of Things (IoT) applications requires intellig...
research
07/14/2021

Memory-Aware Fusing and Tiling of Neural Networks for Accelerated Edge Inference

A rising research challenge is running costly machine learning (ML) netw...
research
10/23/2019

EdgeAI: A Vision for Deep Learning in IoT Era

The significant computational requirements of deep learning present a ma...
research
03/05/2021

Environmental Sound Classification on the Edge: A Pipeline for Deep Acoustic Networks on Extremely Resource-Constrained Devices

Significant efforts are being invested to bring state-of-the-art classif...
research
05/25/2022

Topological Simplification of Signals for Inference and Approximate Reconstruction

As Internet of Things (IoT) devices become both cheaper and more powerfu...

Please sign up or login with your details

Forgot password? Click here to reset