In Teacher We Trust: Learning Compressed Models for Pedestrian Detection

12/01/2016
by   Jonathan Shen, et al.
0

Deep convolutional neural networks continue to advance the state-of-the-art in many domains as they grow bigger and more complex. It has been observed that many of the parameters of a large network are redundant, allowing for the possibility of learning a smaller network that mimics the outputs of the large network through a process called Knowledge Distillation. We show, however, that standard Knowledge Distillation is not effective for learning small models for the task of pedestrian detection. To improve this process, we introduce a higher-dimensional hint layer to increase information flow. We also estimate the variance in the outputs of the large network and propose a loss function to incorporate this uncertainty. Finally, we attempt to boost the complexity of the small network without increasing its size by using as input hand-designed features that have been demonstrated to be effective for pedestrian detection. We succeed in training a model that contains 400× fewer parameters than the large network while outperforming AlexNet on the Caltech Pedestrian Dataset.

READ FULL TEXT

page 6

page 7

page 8

research
04/17/2020

Triplet Loss for Knowledge Distillation

In recent years, deep learning has spread rapidly, and deeper, larger mo...
research
09/20/2019

Learning Lightweight Pedestrian Detector with Hierarchical Knowledge Distillation

It remains very challenging to build a pedestrian detection system for r...
research
09/19/2020

Weight Distillation: Transferring the Knowledge in Neural Network Parameters

Knowledge distillation has been proven to be effective in model accelera...
research
02/14/2023

Multi-teacher knowledge distillation as an effective method for compressing ensembles of neural networks

Deep learning has contributed greatly to many successes in artificial in...
research
04/08/2019

Knowledge Distillation For Recurrent Neural Network Language Modeling With Trust Regularization

Recurrent Neural Networks (RNNs) have dominated language modeling becaus...
research
10/13/2015

Deep convolutional neural networks for pedestrian detection

Pedestrian detection is a popular research topic due to its paramount im...
research
11/19/2017

MicroExpNet: An Extremely Small and Fast Model For Expression Recognition From Frontal Face Images

This paper is aimed at creating extremely small and fast convolutional n...

Please sign up or login with your details

Forgot password? Click here to reset