Designing Network Design Spaces

03/30/2020
by   Ilija Radosavovic, et al.
8

In this work, we present a new network design paradigm. Our goal is to help advance the understanding of network design and discover design principles that generalize across settings. Instead of focusing on designing individual network instances, we design network design spaces that parametrize populations of networks. The overall process is analogous to classic manual design of networks, but elevated to the design space level. Using our methodology we explore the structure aspect of network design and arrive at a low-dimensional design space consisting of simple, regular networks that we call RegNet. The core insight of the RegNet parametrization is surprisingly simple: widths and depths of good networks can be explained by a quantized linear function. We analyze the RegNet design space and arrive at interesting findings that do not match the current practice of network design. The RegNet design space provides simple and fast networks that work well across a wide range of flop regimes. Under comparable training settings and flops, the RegNet models outperform the popular EfficientNet models while being up to 5x faster on GPUs.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/30/2019

On Network Design Spaces for Visual Recognition

Over the past several years progress in designing better neural network ...
research
03/14/2023

Practically Solving LPN in High Noise Regimes Faster Using Neural Networks

We conduct a systematic study of solving the learning parity with noise ...
research
01/18/2019

Machine Learning with Clos Networks

We present a new methodology for improving the accuracy of small neural ...
research
04/02/2019

Exploring Randomly Wired Neural Networks for Image Recognition

Neural networks for image recognition have evolved through extensive man...
research
12/19/2019

MVF Designer: Design and Visualization of Morse Vector Fields

Vector field design on surfaces was originally motivated by applications...
research
08/30/2021

Searching for Two-Stream Models in Multivariate Space for Video Recognition

Conventional video models rely on a single stream to capture the complex...
research
07/18/2012

Communities of Minima in Local Optima Networks of Combinatorial Spaces

In this work we present a new methodology to study the structure of the ...

Please sign up or login with your details

Forgot password? Click here to reset