DeepAI AI Chat
Log In Sign Up

Depth Augmented Networks with Optimal Fine-tuning

03/25/2019
by   Tasfia Shermin, et al.
Federation University Australia
1

Convolutional neural networks (CNN) have been shown to achieve state-of-the-art performance in a significant number of computer vision tasks. Although they require large labelled training datasets to learn the CNN models, they have striking attributes of transferring learned representations from large source sets to smaller target sets by normal fine-tuning approaches. Prior research has shown that these techniques boost the performance on smaller target sets. In this paper, we demonstrate that growing network depth capacity beyond classification layer along with careful normalization and scaling scheme boosts fine-tuning by creating harmony between the pre-trained and new layers to adjust more to the target task. This indicates pre-trained classification layer holds high-level (global) image information that can be propagated through the newly introduced layers in fine-tuning. We evaluate our depth augmented networks following our designed incremental fine-tuning scheme on several benchmark datatsets and show that they outperform contemporary transfer learning approaches. On average, for fine-grained datasets we achieve up to 6.7 (VGG16) improvement than normal fine-tuning. In addition, our in-depth analysis manifests freezing highly generic layers encourage better learning of target tasks. Furthermore, we have found that the learning rate for newly introduced layers of depth augmented networks depend on target set and size of new layers.

READ FULL TEXT
07/18/2019

Growing a Brain: Fine-Tuning by Increasing Model Capacity

CNNs have made an undeniable impact on computer vision through the abili...
01/22/2020

AutoFCL: Automatically Tuning Fully Connected Layers for Transfer Learning

Deep Convolutional Neural Networks (CNN) have evolved as popular machine...
02/15/2023

The Expressive Power of Tuning Only the Norm Layers

Feature normalization transforms such as Batch and Layer-Normalization h...
06/29/2019

NetTailor: Tuning the Architecture, Not Just the Weights

Real-world applications of object recognition often require the solution...
11/06/2016

Beyond Fine Tuning: A Modular Approach to Learning on Small Data

In this paper we present a technique to train neural network models on s...
04/29/2019

ConvTimeNet: A Pre-trained Deep Convolutional Neural Network for Time Series Classification

Training deep neural networks often requires careful hyper-parameter tun...
01/27/2021

CNN with large memory layers

This work is centred around the recently proposed product key memory str...