Theory-Inspired Path-Regularized Differential Network Architecture Search

06/30/2020
by   Pan Zhou, et al.
0

Despite its high search efficiency, differential architecture search (DARTS) often selects network architectures with dominated skip connections which lead to performance degradation. However, theoretical understandings on this issue remain absent, hindering the development of more advanced methods in a principled way. In this work, we solve this problem by theoretically analyzing the effects of various types of operations, e.g. convolution, skip connection and zero operation, to the network optimization. We prove that the architectures with more skip connections can converge faster than the other candidates, and thus are selected by DARTS. This result, for the first time, theoretically and explicitly reveals the impact of skip connections to fast network optimization and its competitive advantage over other types of operations in DARTS. Then we propose a theory-inspired path-regularized DARTS that consists of two key modules: (i) a differential group-structured sparse binary gate introduced for each operation to avoid unfair competition among operations, and (ii) a path-depth-wise regularization used to incite search exploration for deep architectures that often converge slower than shallow ones as shown in our theory and are not well explored during the search. Experimental results on image classification tasks validate its advantages. Codes and models will be released.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/11/2023

Operation-level Progressive Differentiable Architecture Search

Differentiable Neural Architecture Search (DARTS) is becoming more and m...
research
05/07/2020

Noisy Differentiable Architecture Search

Simplicity is the ultimate sophistication. Differentiable Architecture S...
research
11/27/2019

Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

Differential Architecture Search (DARTS) is now a widely disseminated we...
research
09/02/2020

DARTS-: Robustly Stepping out of Performance Collapse Without Indicators

Despite the fast development of differentiable architecture search (DART...
research
04/10/2022

Enhancing the Robustness, Efficiency, and Diversity of Differentiable Architecture Search

Differentiable architecture search (DARTS) has attracted much attention ...
research
09/09/2019

LCSCNet: Linear Compressing Based Skip-Connecting Network for Image Super-Resolution

In this paper, we develop a concise but efficient network architecture c...
research
02/20/2021

Towards Accurate and Compact Architectures via Neural Architecture Transformer

Designing effective architectures is one of the key factors behind the s...

Please sign up or login with your details

Forgot password? Click here to reset