PICO: Pipeline Inference Framework for Versatile CNNs on Diverse Mobile Devices

06/17/2022
by   Xiang Yang, et al.
0

Recent researches in artificial intelligence have proposed versatile convolutional neural networks (CNN) with different structures and substantially improved the accuracy of various intelligent applications. Nevertheless, the CNN inference imposes heavy computation overhead on mobile devices, but uploading the large volume of raw data to the cloud causes significant network latency. Motivated by the spatial independence of convolution operation, we propose pipeline cooperation (PICO) framework to accelerate CNN inference using multiple diverse mobile devices in this paper. PICO divides the CNN and mobile devices into several stages and combines them into an inference pipeline. PICO faces three main challenges: (1) Parallelizing the convolution operation introduces redundant calculation. (2) The partition is greatly complicated since the structures of many CNNs are directed acyclic graphs (DAG). (3) The mobile devices own diverse computing resources. In response to these issues, a two-step optimization is proposed based on deep analysis. We first orchestrate the DAG into sequential pieces, then divides these pieces and devices into stages. The optimization goal is to minimize the redundant calculation during partition and maximize the throughput. In our experiment with 2 ∼ 8 RaspberryPi devices, the throughput can be improved by 1.8 ∼ 6.8 × under different CPU frequencies.

READ FULL TEXT
research
02/15/2019

AutoQB: AutoML for Network Quantization and Binarization on Mobile Devices

In this paper, we propose a hierarchical deep reinforcement learning (DR...
research
09/10/2019

Characterizing the Deep Neural Networks Inference Performance of Mobile Applications

Today's mobile applications are increasingly leveraging deep neural netw...
research
07/04/2017

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

We introduce an extremely computation-efficient CNN architecture named S...
research
04/08/2019

Temporal Convolution for Real-time Keyword Spotting on Mobile Devices

Keyword spotting (KWS) plays a critical role in enabling speech-based us...
research
06/21/2023

Adaptive DNN Surgery for Selfish Inference Acceleration with On-demand Edge Resource

Deep Neural Networks (DNNs) have significantly improved the accuracy of ...
research
09/27/2017

Modeling the Resource Requirements of Convolutional Neural Networks on Mobile Devices

Convolutional Neural Networks (CNNs) have revolutionized the research in...
research
08/04/2016

Faster CNNs with Direct Sparse Convolutions and Guided Pruning

Phenomenally successful in practical inference problems, convolutional n...

Please sign up or login with your details

Forgot password? Click here to reset