Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives

03/24/2020
by   Duo Li, et al.
0

While the depth of modern Convolutional Neural Networks (CNNs) surpasses that of the pioneering networks with a significant margin, the traditional way of appending supervision only over the final classifier and progressively propagating gradient flow upstream remains the training mainstay. Seminal Deeply-Supervised Networks (DSN) were proposed to alleviate the difficulty of optimization arising from gradient flow through a long chain. However, it is still vulnerable to issues including interference to the hierarchical representation generation process and inconsistent optimization objectives, as illustrated theoretically and empirically in this paper. Complementary to previous training strategies, we propose Dynamic Hierarchical Mimicking, a generic feature learning mechanism, to advance CNN training with enhanced generalization ability. Partially inspired by DSN, we fork delicately designed side branches from the intermediate layers of a given neural network. Each branch can emerge from certain locations of the main branch dynamically, which not only retains representation rooted in the backbone network but also generates more diverse representations along its own pathway. We go one step further to promote multi-level interactions among different branches through an optimization formula with probabilistic prediction matching losses, thus guaranteeing a more robust optimization process and better representation ability. Experiments on both category and instance recognition tasks demonstrate the substantial improvements of our proposed method over its corresponding counterparts using diverse state-of-the-art CNN architectures. Code and models are publicly available at https://github.com/d-li14/DHM

READ FULL TEXT
research
06/03/2019

Deeply-supervised Knowledge Synergy

Convolutional Neural Networks (CNNs) have become deeper and more complic...
research
11/21/2020

DmifNet:3D Shape Reconstruction Based on Dynamic Multi-Branch Information Fusion

3D object reconstruction from a single-view image is a long-standing cha...
research
11/28/2019

Transform-Invariant Convolutional Neural Networks for Image Classification and Search

Convolutional neural networks (CNNs) have achieved state-of-the-art resu...
research
01/24/2017

Training Group Orthogonal Neural Networks with Privileged Information

Learning rich and diverse representations is critical for the performanc...
research
06/26/2021

Interflow: Aggregating Multi-layer Feature Mappings with Attention Mechanism

Traditionally, CNN models possess hierarchical structures and utilize th...
research
01/12/2022

Structure and position-aware graph neural network for airway labeling

We present a novel graph-based approach for labeling the anatomical bran...
research
07/13/2022

Eliminating Gradient Conflict in Reference-based Line-art Colorization

Reference-based line-art colorization is a challenging task in computer ...

Please sign up or login with your details

Forgot password? Click here to reset