MaskConnect: Connectivity Learning by Gradient Descent

07/28/2018
by   Karim Ahmed, et al.
0

Although deep networks have recently emerged as the model of choice for many computer vision problems, in order to yield good results they often require time-consuming architecture search. To combat the complexity of design choices, prior work has adopted the principle of modularized design which consists in defining the network in terms of a composition of topologically identical or similar building blocks (a.k.a. modules). This reduces architecture search to the problem of determining the number of modules to compose and how to connect such modules. Again, for reasons of design complexity and training cost, previous approaches have relied on simple rules of connectivity, e.g., connecting each module to only the immediately preceding module or perhaps to all of the previous ones. Such simple connectivity rules are unlikely to yield the optimal architecture for the given problem. In this work we remove these predefined choices and propose an algorithm to learn the connections between modules in the network. Instead of being chosen a priori by the human designer, the connectivity is learned simultaneously with the weights of the network by optimizing the loss function of the end task using a modified version of gradient descent. We demonstrate our connectivity learning method on the problem of multi-class image classification using two popular architectures: ResNet and ResNeXt. Experiments on four different datasets show that connectivity learning using our approach yields consistently higher accuracy compared to relying on traditional predefined rules of connectivity. Furthermore, in certain settings it leads to significant savings in number of parameters.

READ FULL TEXT

page 12

page 22

research
09/27/2017

Connectivity Learning in Multi-Branch Networks

While much of the work in the design of convolutional networks over the ...
research
06/03/2019

Discovering Neural Wirings

The success of neural networks has driven a shift in focus from feature ...
research
11/18/2019

Fine-Grained Neural Architecture Search

We present an elegant framework of fine-grained neural architecture sear...
research
04/16/2019

SparseMask: Differentiable Connectivity Learning for Dense Image Prediction

In this paper, we aim at automatically searching an efficient network ar...
research
11/23/2017

Deep Expander Networks: Efficient Deep Networks from Graph Theory

Deep Neural Networks, while being unreasonably effective for several vis...
research
05/11/2022

Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis

Advanced deep neural networks (DNNs), designed by either human or AutoML...

Please sign up or login with your details

Forgot password? Click here to reset