EARLIN: Early Out-of-Distribution Detection for Resource-efficient Collaborative Inference

by   Sumaiya Tabassum Nimi, et al.

Collaborative inference enables resource-constrained edge devices to make inferences by uploading inputs (e.g., images) to a server (i.e., cloud) where the heavy deep learning models run. While this setup works cost-effectively for successful inferences, it severely underperforms when the model faces input samples on which the model was not trained (known as Out-of-Distribution (OOD) samples). If the edge devices could, at least, detect that an input sample is an OOD, that could potentially save communication and computation resources by not uploading those inputs to the server for inference workload. In this paper, we propose a novel lightweight OOD detection approach that mines important features from the shallow layers of a pretrained CNN model and detects an input sample as ID (In-Distribution) or OOD based on a distance function defined on the reduced feature space. Our technique (a) works on pretrained models without any retraining of those models, and (b) does not expose itself to any OOD dataset (all detection parameters are obtained from the ID training dataset). To this end, we develop EARLIN (EARLy OOD detection for Collaborative INference) that takes a pretrained model and partitions the model at the OOD detection layer and deploys the considerably small OOD part on an edge device and the rest on the cloud. By experimenting using real datasets and a prototype implementation, we show that our technique achieves better results than other approaches in terms of overall accuracy and cost when tested against popular OOD datasets on top of popular deep learning models pretrained on benchmark datasets.


page 1

page 2

page 3

page 4


AppealNet: An Efficient and Highly-Accurate Edge/Cloud Collaborative Architecture for DNN Inference

This paper presents AppealNet, a novel edge/cloud collaborative architec...

Hazard Detection in Supermarkets using Deep Learning on the Edge

Supermarkets need to ensure clean and safe environments for both shopper...

Knowledge Transfer For On-Device Speech Emotion Recognition with Neural Structured Learning

Speech emotion recognition (SER) has been a popular research topic in hu...

DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices

Recent years have witnessed the great success of vision transformer (ViT...

An Efficient Split Fine-tuning Framework for Edge and Cloud Collaborative Learning

To enable the pre-trained models to be fine-tuned with local data on edg...

DE-CROP: Data-efficient Certified Robustness for Pretrained Classifiers

Certified defense using randomized smoothing is a popular technique to p...

Complexity-aware Adaptive Training and Inference for Edge-Cloud Distributed AI Systems

The ubiquitous use of IoT and machine learning applications is creating ...

Please sign up or login with your details

Forgot password? Click here to reset