DeepAI AI Chat
Log In Sign Up

In Teacher We Trust: Learning Compressed Models for Pedestrian Detection

by   Jonathan Shen, et al.
Carnegie Mellon University

Deep convolutional neural networks continue to advance the state-of-the-art in many domains as they grow bigger and more complex. It has been observed that many of the parameters of a large network are redundant, allowing for the possibility of learning a smaller network that mimics the outputs of the large network through a process called Knowledge Distillation. We show, however, that standard Knowledge Distillation is not effective for learning small models for the task of pedestrian detection. To improve this process, we introduce a higher-dimensional hint layer to increase information flow. We also estimate the variance in the outputs of the large network and propose a loss function to incorporate this uncertainty. Finally, we attempt to boost the complexity of the small network without increasing its size by using as input hand-designed features that have been demonstrated to be effective for pedestrian detection. We succeed in training a model that contains 400× fewer parameters than the large network while outperforming AlexNet on the Caltech Pedestrian Dataset.


page 6

page 7

page 8


Triplet Loss for Knowledge Distillation

In recent years, deep learning has spread rapidly, and deeper, larger mo...

Learning Lightweight Pedestrian Detector with Hierarchical Knowledge Distillation

It remains very challenging to build a pedestrian detection system for r...

Weight Distillation: Transferring the Knowledge in Neural Network Parameters

Knowledge distillation has been proven to be effective in model accelera...

Multi-teacher knowledge distillation as an effective method for compressing ensembles of neural networks

Deep learning has contributed greatly to many successes in artificial in...

Deep convolutional neural networks for pedestrian detection

Pedestrian detection is a popular research topic due to its paramount im...

Solvable Model for Inheriting the Regularization through Knowledge Distillation

In recent years the empirical success of transfer learning with neural n...

Smaller3d: Smaller Models for 3D Semantic Segmentation Using Minkowski Engine and Knowledge Distillation Methods

There are various optimization techniques in the realm of 3D, including ...

Code Repositories