Diving deeper into mentee networks

by   Ragav Venkatesan, et al.

Modern computer vision is all about the possession of powerful image representations. Deeper and deeper convolutional neural networks have been built using larger and larger datasets and are made publicly available. A large swath of computer vision scientists use these pre-trained networks with varying degrees of successes in various tasks. Even though there is tremendous success in copying these networks, the representational space is not learnt from the target dataset in a traditional manner. One of the reasons for opting to use a pre-trained network over a network learnt from scratch is that small datasets provide less supervision and require meticulous regularization, smaller and careful tweaking of learning rates to even achieve stable learning without weight explosion. It is often the case that large deep networks are not portable, which necessitates the ability to learn mid-sized networks from scratch. In this article, we dive deeper into training these mid-sized networks on small datasets from scratch by drawing additional supervision from a large pre-trained network. Such learning also provides better generalization accuracies than networks trained with common regularization techniques such as l2, l1 and dropouts. We show that features learnt thus, are more general than those learnt independently. We studied various characteristics of such networks and found some interesting behaviors.


A Light in the Dark: Deep Learning Practices for Industrial Computer Vision

In recent years, large pre-trained deep neural networks (DNNs) have revo...

Pre-Trained Convolutional Neural Network Features for Facial Expression Recognition

Facial expression recognition has been an active area in computer vision...

Arithmetic addition of two integers by deep image classification networks: experiments to quantify their autonomous reasoning ability

The unprecedented performance achieved by deep convolutional neural netw...

UMDFaces: An Annotated Face Dataset for Training Deep Networks

Recent progress in face detection (including keypoint detection), and re...

Surface Masked AutoEncoder: Self-Supervision for Cortical Imaging Data

Self-supervision has been widely explored as a means of addressing the l...

Training Efficient CNNS: Tweaking the Nuts and Bolts of Neural Networks for Lighter, Faster and Robust Models

Deep Learning has revolutionized the fields of computer vision, natural ...

Statistical transformer networks: learning shape and appearance models via self supervision

We generalise Spatial Transformer Networks (STN) by replacing the parame...

Please sign up or login with your details

Forgot password? Click here to reset