XNAS: Neural Architecture Search with Expert Advice

06/19/2019 ∙ by Niv Nayman, et al. ∙ 3

This paper introduces a novel optimization method for differential neural architecture search, based on the theory of prediction with expert advice. Its optimization criterion is well fitted for an architecture-selection, i.e., it minimizes the regret incurred by a sub-optimal selection of operations. Unlike previous search relaxations, that require hard pruning of architectures, our method is designed to dynamically wipe out inferior architectures and enhance superior ones. It achieves an optimal worst-case regret bound and suggests the use of multiple learning-rates, based on the amount of information carried by the backward gradients. Experiments show that our algorithm achieves a strong performance over several image classification datasets. Specifically, it obtains an error rate of 1.6 settings, and achieves state-of-the-art results on three additional datasets.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In recent years tremendous efforts have been put into a manual design of high performance neural networks 

larsson2016fractalnet ; hu2018squeeze ; szegedy2016rethinking_label_smooth ; szegedy2015going . An emerging alternative approach is replacing the manual design with automated Neural Architecture Search (NAS

). NAS excels in finding architectures which yield state-of-the-art results. Earlier NAS works were based on reinforcement learning 

zoph2016neural ; NASNET , sequential optimization PNAS

, and evolutionary algorithms 

Real18Regularized , and required immense computational resources, sometimes demanding years of GPU compute time in order to output an architecture. More recent NAS methods reduce the search time significantly, e.g. via weight-sharing ENAS or by a continuous relaxation of the space liu2018darts , making the search affordable and applicable to real problems.

While current NAS methods provide encouraging results, they still suffer from several shortcomings. For example, a large number of hyper-parameters that are not easy to tune, hard pruning decisions that are performed sub-optimally at once at the end of the search, and a weak theoretical understanding. This cultivates skepticism and criticism of the utility of NAS in general. Some recent works even suggest that current search methods are only slightly better than random search and further imply that some selection methods are not well principled and are basically random li2019random ; sciuto2019evaluating .

To provide more principled methods, we view NAS as an online selection task, and rely on Prediction with Experts Advice (PEA) theory cesa2006prediction for the selection. Our key contribution is the introduction of XNAS (eXperts Neural Architecture Search), an optimization method (section 2.2) that is well suited for optimizing inner architecture weights over a differentiable architecture search space (section 2.1). We propose a setup in which the experts represent inner neural operations and connections, whose dominance is specified by architecture weights.

Our proposed method addresses the mentioned shortcomings of current NAS methods. For the mitigation of the hard pruning, we leverage the Exponentiated-Gradient (EG) algorithm kivinen1997exponentiated

, which favors sparse weight vectors to begin with, enhanced by a wipeout mechanism for dynamically pruning inferior experts during the search process. Additionally, the algorithm requires less hyper-parameters to be tuned (

section 3.2.2), and the theory behind it further provides guidance for the choice of learning rates. Specifically, the algorithm avoids the decay of architecture weights goodfellow2016deep , which is shown to promote selection of arbitrary architectures.

Additionally, XNAS features several desirable properties, such as achieving an optimal worst-case regret bound (section 3.1) and suggesting to assign different learning rates for different groups of experts. Considering an appropriate reward term, the algorithm is more robust to the initialization of the architecture weights and inherently enables the recovery of ’late bloomers’, i.e., experts which may become effective only after a warm-up period (section 3.2.1). The wipeout mechanism allows the recovery of experts with a chance of being selected at the end of the process.

We compare XNAS to previous methods and demonstrate its properties and effectiveness over statistical and deterministic setups, as well as over public datasets (section 4). It achieves state-of-the-art performance over datasets, and top-NAS over rest, with significant improvements. For example, XNAS reaches error over CIFAR-10, more than improvement over existing NAS methods.

2 Proposed Approach

To lay out our approach we first reformulate the differentiable architecture search space of DARTS liu2018darts in a way that enables direct optimization over the architecture weights. We then propose a novel optimizer that views NAS as an online selection task, and relies on PEA theory for the selection.

2.1 Neural Architecture Space

We start with a brief review of the PEA settings and then describe our view of the search space as separable PEA sub-spaces. This enables us to leverage PEA theory for NAS.

PEA Settings. PEA cesa2006prediction refers to a sequential decision making framework, dealing with a decision maker, i.e. a forecaster, whose goal is to predict an unknown outcome sequence while having access to a set of experts’ advises, i.e. predictions. Denote the experts’ predictions at time by , where is the decision space, which we assume to be a convex subset of a vector space. Denote the forecaster’s prediction

, and a non-negative loss function

. At each time step , the forecaster observes and predicts . The forecaster and the experts suffer losses of and respectively.

The Search Space Viewed as Separable PEA Sub-spaces. We view the search space suggested by DARTS liu2018darts as multiple separable sub-spaces of experts, as illustrated in Figure 1, described next. An architecture is built from replications of normal and reduction cells represented as a directed acyclic graph. Every node in the graph represents a feature map and each directed edge is associated with a forecaster, that predicts a feature map given the input . Intermediate nodes are computed based on all of their predecessors: . The output of the cell is obtained by applying a reduction operation (e.g. concatenation) to the intermediate nodes. During the search stage, every forecaster combines experts’ feature map predictions forming its own prediction,


From now on, we will ignore the superscript indices

for brevity. Each expert represents a neural operation, e.g. convolution or max pooling, associated with network weights

, that receives an input at time , and outputs a feature map prediction. Thus a time index is attached, as each prediction is associated with updated weights .

Our architecture search approach is composed of two stages. In the search stage, the weights and are alternately optimized as described in section 2.2; then, in the discretization stage, a discrete child architecture is obtained as explained next.

The Discretization Stage. Once the architecture weights are optimized, the final discrete neural architecture is obtained by performing the following discretization stage, adopted from liu2018darts : Firstly, the strongest two predecessor edges are retained for each intermediate node. The strength of an edge is defined as . Lastly, every forecaster is replaced by the corresponding strongest expert.

2.2 XNAS: eXperts Neural Architecture Search

The differential space, described in section 2.1, enables direct optimization over the architecture weights via gradient-descent based techniques. Previous methods adopted generic optimizers commonly used for training the network weights. For example liu2018darts ; xie2018snas ; chen2019progressive ; casale2019probabilistic used adam kingma2014adam , and noy2019asap used SGD with momentum. While those optimizers excel in joint minimization of neural network losses when applied to network weights, NAS is a essentially a selection task, aiming to select a subset of experts out of a superset. The experts weights form a convex combination, as they compete over a forecaster’s attention.

We argue that a generic alternate optimization of network weights and architecture weights, as suggested in previous works, e.g. hundt2019sharpdarts ; liu2018darts , is not suitable for the unique structure of the architecture space. Hence, we design a tailor-made optimizer for this task, inspired by PEA theory. In order to evaluate experts’ performance, a loss is to be associated with each expert. However, an explicit loss is not assigned to each expert, as opposed to a back-propagated loss gradient. Therefore, we base our algorithm on a version of the Exponentiated-Gradient (EG) algorithm adapted for the NAS space. EG algorithms favor sparse weight vectors kivinen1997exponentiated , thus fit well to online selection problems.

We introduce the XNAS (eXperts Neural Architecture Search) algorithm, outlined in Algorithm 1 for a single forecaster. XNAS alternates between optimizing the network weights and the architecture weights in a designated manner. After updating (line 4) by descending the train loss , the forecaster makes a prediction based on the mixture of experts (line 5), then are updated with respect to the validation loss according to the EG rule (lines 8-9). Next, the optimizer wipes-out weak experts (lines 11-12), effectively assigning their weights to the remaining ones. The exponential update terms, i.e. rewards, are determined by the projection of the loss gradient on the experts’ predictions: . In section 3.1 we relate to the equivalence of this reward to the one associated with policy gradient search williams1992simple applied to NAS.

1:  Input: The learning rate , Loss-gradient bound , Experts predictions
2:  Init:
3:  for rounds  do
4:     Update by descending
5:               #Predict
7:     for  do
8:             #Rewards
9:            #EG step
10:     end for
12:             #Wipeout
13:  end for
Algorithm 1 XNAS for a single forecaster
Figure 1: A visual illustration of the search space as separable experts sub-problems, equipped with XNAS forward and backward propagation. forward and backward passes are presented as grey and orange arrows respectively.

The purpose of the wipeout step is threefold. First, the removal of weak experts consistently smooths the process towards selecting a final architecture at the descretization stage described in section 2.1. Thus it mitigates the harsh final pruning of previous methods, which results in a relaxation bias addressed in  snas ; noy2019asap . Second, it dynamically reduces the number of network weights, thus simplifying the optimization problem, avoiding over-fitting and allowing it to converge to a better solution. Last, it speeds-up the architecture search, as the number of graph computations decreases with the removal of experts.

3 Analysis and Discussion

3.1 Theoretical Analysis

In this section we analyse the performance of the proposed algorithm. For this purpose, we introduce the regret as a performance measure for NAS algorithms. Showing that the wipeout mechanism cannot eliminate the best expert (lemma 1), we provide theoretical guarantees for XNAS with respect to that measure (theorem 1). Proofs appear in Section 7 of the supplementary material for brevity. Relaying on the theoretical analysis, we extract practical instructions with regard to the choice of multiple learning rates. Finally we briefly discuss the equivalence of our reward to the one considered by the policy gradient approach applied to NAS.

The Regret as a Performance Measure. Denote the regret, the cumulative losses of the forecaster and of the th expert at time by,


respectively. The regret measures how much the forecaster regrets not following the advice of the best expert in hindsight. This criterion suits our setup as we optimize a mixture of experts and select the best one by the end of the process.

In classical learning theory, statistical properties of the underlying process may be estimated on the basis of stationarity assumptions over the sequence of past observations. Thus effective prediction rules can be derived from these estimates 

cesa2006prediction . However, NAS methods that alternately learn the architecture weights and train the network weights are highly non-stationary. In PEA theory, no statistical assumptions are made, as “simplicity is a merit” hazan2016introduction , and worst-case bounds are derived for the forecaster’s performance. We obtain such bounds for the wipeout mechanism and the regret.

A Safe Wipeout. In XNAS (line 11), by the choice of the wipeout thresholds, experts with no chance of taking the lead by the end of the search are wiped out along the process. In a worse-case setup, a single incorrect wipeout might result in a large regret, i.e., linear in the number of steps , due to a loss gap at each consecutive step. The following lemma assures that this cannot happen,

Lemma 1.

In XNAS, the optimal expert in hindsight cannot be wiped out.

The wipeout effectively transfers the attention to leading experts. Define the wipeout factor and the aggregated wipeout factor as and , respectively.

Lemma 2.

The aggregated wipeout factor satisfies .

Equipped with both lemmas, we show that the wipeout may improve the EG regret bound for certain reward sequences.

Regret Bounds. Our main theorem guarantees an upper bound for the regret,

Theorem 1 (XNAS Regret Bound).

The regret of the XNAS algorithm 1, with experts and learning rate , incurring a sequence of non-negative convex losses of -bounded rewards, satisfies,


As an input parameter of XNAS, the learning rate cannot be determined based on the value of , since the later depends on the data sequence. Choosing the minimizer of the first two terms of (3) fully known in advance, yields the following tight upper bound,


The regret upper bound of XNAS is tight, as the lower bound can be shown to be of  haussler1995tight . In addition, the wipeout related term reduces the regret in an amount which depends on the data sequences through , as it effectively contributes the attention of weak experts to the leading ones. For comparison, under the same assumptions, the worst-case regret bound of gradient-descent is of  hazan2016introduction , while the one of Adam is linear in  reddi2019convergence . An illustration of the relationship between the regret and the rate of correct expert selection appears in section 8.3 of the supplementary material, where XNAS is shown to achieve a better regret compared to a generic optimizer.

Multiple Learning Rates. Equation 4 connects the optimal theoretical learning rate with the number of steps , which is also the number of gradient feedbacks received by the experts. Since forecasters weights are being replicated among different cells, the number of feedbacks is different for normal and reduction cells (section 2.1). Explicitly, , where are the effective horizon

, the validation set size, the number of epochs and the number of replications for cell type

respectively. We adopt the usage of multiple learning rates in our experiments as upper bounds on the learning rates for minimizing the upper bound of the regret.

The Connection to Policy Gradient. We conclude this section by pointing out an interesting connection between policy gradient in NAS zoph2016neural and PEA theory. We refer to the PEA based reward term in line 8 of algorithm 1. This reward has been shown by snas to be the same effective reward of a policy gradient method applied to the common NAS optimization criterion zoph2016neural ; NASNET ; ENAS

. More precisely, consider the case where instead of mixing the experts’ predictions, the forecaster is to sample a single expert at each step with probability

, specified in (1). Then the effective reward associated with the policy gradient will be exactly the derived PEA reward, . XNAS optimizes with respect to the same reward, while avoiding the sampling inefficiency associated with policy gradient methods.

3.2 Key Properties and Discussion

In this section we discuss some of the key properties of XNAS. For each of this properties we provide supporting derivations, illustrations and demonstrations appearing in section 8 of the supplementary material for brevity.

3.2.1 The Recovery of Late Bloomers and Robustness to Initialization

In this section we point out a key difference between our proposed update rule and the one used in previous works. We refer to Gradient Descent (GD) with softmax as updating the parameters , by descending respectively. Variants of GD with softmax, as used in liu2018darts to optimize the architecture weights, suppress operations that are weak at the initial iterations, making it more difficult for them to “bloom” and increase their weights. This could be problematic, e.g. in the two following cases. First, consider the best expert starting with a poor weight which gradually rises. This could be the case when an expert representing a parameterized operation (e.g. a convolutional layer) competes with an unparameterized one (e.g. a pooling layer), as the first requires some period for training, as stated by noy2019asap ; hundt2019sharpdarts . Second, consider a noisy setup, where the best expert in hindsight could receive some hard penalties before other inferior experts do. In NAS we deal with stochastic settings associated with the training data.

We inspect the update term of GD with softmax,


Hence, the effective reward in this case is,


See derivations in section 8.4. The linear dependence on the expert’s weight in (6) implies that GD with softmax makes it harder for an expert whose weight is weak at some point to recover and become dominant later on, as the associated rewards are attenuated by the weak expert’s weight.

XNAS mitigates this undesirable behavior. Since for XNAS the update term (8) depends on the architecture weights only indirectly, i.e. through the prediction, the recovery of late bloomers is not discouraged, as demonstrated in section 8.1 of the supplementary material. From the very same reasons, XNAS is more robust to the initialization scheme compared to GD with softmax and its variants, as demonstrated in section 8.2 of the supplementary material. These advantages make XNAS more suitable for the NAS setup.

Note that while the XNAS enables the recovery of experts with badly initialized weights or with delayed rewards, the wipeout mechanism prevents inferior operations that start blooming too late from interfering, by eliminating experts with no chance of leading at the end.

Wipeout Factor. As mentioned in section 2.2, the wipeout mechanism contributes to both optimization process and search duration. A further reduction in duration can be achieved when the wipe-out threshold in line 11 of Algorithm 1 is relaxed with a parameter , being replaced by . This will lead to a faster convergence to a single architecture, with the price of a violation of the theoretical regret. As worst-case bounds tend to be over pessimistic, optimizing over could lead to improved results. We leave that for future work.

3.2.2 Fewer Hyper Parameters

The view of the differentiable NAS problem as an optimization problem solved by variants of GD, e.g. Adam, introduces some common techniques for such schemes along with their corresponding hyper-parameters. Tuning these complicates the search process - the fewer hyper-parameters the better. We next discuss how XNAS simplifies and reduces the number of hyper-parameters.

Theoretically Derived Learning Rates. The determination of the learning rate has a significant impact on the convergence of optimization algorithms. Various scheduling schemes come up, e.g. loshchilov2016sgdr ; smith2017cyclical , as the later additionally suggests a way for obtaining an empirical upper bound on the learning rate. In section 3.1, multiple learning rates are suggested for minimizing the regret bound (4), as

represents normal and reduction cells respectively. For example, for CIFAR10 with 50%:50% train-validation split, 50 search epochs, gradient clipping of

, normal cells and reduction cells both of experts for each forecaster, (4) yields 7.5e-4 and 1.3e-3.

Remark 1.

Note that the proposed learning rates minimize an upper bound of the regret (4) in the case of no wipeout, i.e. the worst case, as the extent of the wipeout cannot be known in advance. Hence the proposed learning rate provides an upper bound on the optimal learning rates and can be further fine-tuned.

Figure 2: Mean normalized entropy vs weight decay. The red dot refers to DARTS’ settings.

No Weight Decay.

Another common technique involving hyper-parameters is weight decay, which has no place in the theory behind XNAS. We claim that the obviation of weight decay by XNAS makes sense. Regularization techniques, such as weight decay, reduce over-fitting of over-parametrized models when applied to these parameters

goodfellow2016deep . No such effect is incurred when applying weight decay on the architecture parameters as they do not play the same role as the trained network parameters . Instead, weight decay encourages uniform dense solutions, as demonstrated in Figure 3.2.2, where the mean normalized entropy increases with the weight decay coefficient. The calculation of the mean normalized entropy is detailed in section 8.5 of the supplementary material. This observation could be associated with the suggestion of recent works li2019random ; sciuto2019evaluating that current search methods are only slightly better than random search. The density of results in a harder degradation in performance once discretization stage occurs (section 2.1), hence sparse solutions are much preferred over dense ones.

No Momentum. The theory behind XNAS obviates momentum qian1999momentum and ADAM’s exponentially decay rates kingma2014adam . Since momentum requires more state variables and more computations, the resulting XNAS optimizer turns out to be simpler, faster and with a smaller memory footprint, compared to commonly used optimizers for NAS, e.g. ADAM liu2018darts ; xie2018snas ; chen2019progressive ; casale2019probabilistic and SGD with momentum noy2019asap .

4 Experiments and Results

In this section we will test XNAS on common image classification benchmarks, and show its effectiveness compared to the other state-of-the-art models.

We used the CIFAR-10 dataset for the main search and evaluation phase. In addition, using the cell found on CIFAR-10 we did transferability experiments on the well-known benchmarks ImageNet, CIFAR-100, SVHN, Fashion-MNIST, Freiburg and CINIC10.

4.1 Architecture Search on CIFAR-10

Using XNAS, we searched on CIFAR-10 in a small parent network for convolutional cells. Then we built a larger network by stacking the learned cells, trained it on CIFAR-10 and compared the results against other NAS methods.

We created the parent network by stacking cells with ordered nodes, each of which connected via forecasters to all previous nodes in the cell and also to the two previous cells outputs. Each forecaster contains seven operations: 3x3 and 5x5 separable and dilated separable convolutions, 3x3 max-pooling, 3x3 average-pooling and an identity. A cells output is a concatenation of the outputs of the four cells nodes.

The search phase lasts epochs. We use the first-order approximation liu2018darts , relating to and as independent parameters which can be optimized separately. The train set is divided into two parts of equal sizes: one is used for training the operations weights and the other for training the architecture weights . With a batch size of , one epoch takes minutes in average on a single GPU111Experiments were performed using a NVIDIA GTX 1080Ti GPU., summing up to hours in total for a single search. Figure 9 shows our learned normal and reduction cells, respectively.

4.2 CIFAR-10 Evaluation Results

We built the evaluation network by stacking cells: normal cells and reduction cells. The reduction cells are placed after and of the network. After each reduction cell we double the amount of channels in the network. We trained the network for epochs using a batch size of

and SGD optimizer with nesterov-momentum. Our learning rate regime was composed of

cycles of power cosine annealing learning rate hundt2019sharpdarts , with amplitude decay factor of per cycle. For regularization we used cutout devries2017improved , scheduled drop-path larsson2016fractalnet , auxiliary towers szegedy2015going , label smoothing szegedy2016rethinking_label_smooth AutoAugment cubuk2018autoaugment and weight decay. To understand the effect of the network size on the final accuracy, we chose to test architecture configurations, XNAS-Small, XNAS-Medium and XNAS-Large, with , and initial network channels respectively. Table 1 shows the performance of XNAS architectures compared to other state-of-the-art NAS methods. We can see from Table 1 that XNAS smallest network variant, XNAS-Small, outperforms on CIFAR-10 previous NAS methods by a large margin. Our largest network variant, XNAS-Large, is the second highest reported score on CIFAR-10 (without additional pre-train data), while having times less parameters than top one cubuk2018autoaugment . In addition, XNAS is among the fastest NAS methods.
CIFAR-10 Architecture Test error Params (M) Search cost AutoAugment cubuk2018autoaugment NAONet-WS NAO PNAS PNAS Amoeba-A Real18Regularized DSO-NAS DSO DARTS(1nd) liu2018darts ENAS ENAS SNAS snas DARTS(2nd) liu2018darts NASNet-A NASNET Amoeba-B Real18Regularized PDARTS chen2019progressive NAONet NAO ProxylessNAS cai2018proxylessnas 10 ASAP + ET noy2019asap SharpDarts hundt2019sharpdarts XNAS-Small XNAS-Medium XNAS-Large Table 1: Classification errors of XNAS compared to state-of-the-art methods on CIFAR-10. The Search cost is measured in GPU days. Test error refers to top-1 test error (%). XNAS Small, Medium and Large refers to network architectures with XNAS cell and , and initial channels respectively.

4.3 Transferability Evaluation

Using the cell found by XNAS search on CIFAR-10, we preformed transferability tests on popular classification benchmarks: ImageNet, CIFAR-100, Fashion-MNIST, SVHN, Freiburg and CINIC10. ImageNet Results. Our ImageNet network was composed of stacked XNAS cells, with two initial stem cells for downscaling. We used initial channels so the total number of network FLOPs is below 600[M], similar to other ImageNet architectures with small computation regime xie2019exploring . We trained the network for epochs with one cycle of power cosine learning rate and a nesterov-momentum optimizer. The results are presented in Table 2. We can see from Table 2 that XNAS transferability results on ImageNet are highly competitive, outperforming all previous NAS cells. ImageNet Architecture Test error Params (M) Search cost SNAS snas ASAP noy2019asap DARTS liu2018darts NASNet-A NASNET PNAS PNAS Amoeba-A Real18Regularized RandWire xie2019exploring SharpDarts hundt2019sharpdarts Amoeba-C Real18Regularized XNAS 0.3 Table 2: Transferability classification error of XNAS, compared to top NAS cells, on ImageNet. Test error refers to top-1 test error (%). Search cost is measured in GPU days.

Additional Results. We further tested XNAS transferability abilities on smaller datasets: CIFAR-100 cifar100 , Fashion-MNIST fashionMnist , SVHN SVHN , Freiburg Freiburg and CINIC10 darlow2018cinic . We chose to use the XNAS-Small architecture, with similar training scheme to the one described in section 4.2. Table 3 shows the performance of our model compared to NAS methods. We can see that XNAS cell excels on the datasets tested. On CIFAR-100 it surpasses the next top cell by %, achieving the second highest reported score on CIFAR-100 (without additional pre-train data), second only to cubuk2018autoaugment . On Fashion-MNIST, Freiburg and CINIC10, to the best of our knowledge XNAS achieves a new state-of-the-art accuracy.



CIFAR100 Error FMNIST Error SVHN Error Freiburg Error CINIC10 Error Params
(M) Search
Known SotA cubuk2018autoaugment zhong2017random cubuk2018autoaugment noy2019asap noy2019asap 26 cubuk2018autoaugment -
PDARTS chen2019progressive - - - -
NAONet-1 NAO - - - -
NAONet-2 NAO - - - -
PDARTS-L chen2019progressive - - - -
SNAS snas
Amoeba-A Real18Regularized
DARTS liu2018darts
ASAP noy2019asap
XNAS-Small 0.3
Table 3: Classification errors of XNAS, compared to state-of-the-art NAS methods, on several datasets. Error refers to top-1 test error (%). The Search cost is measured in GPU days. Results marked with are taken from noy2019asap , which tested and compared different cells on various datasets.

5 Related Work

Mentions of Experts in deep learning 

rasmussen2002infinite ; yao2009hierarchical ; garmash2016ensemble ; aljundi2017expert literature go decades back Jacobs:1991 ; chen1999improved , typically combining models as separable experts sub-models. A different concept of using multiple mixtures of experts as inner parts of a deep model, where each mixture has its own gating network, is presented in eigen2013learning . Following works build upon this idea and include a gating mechanism per mixture CondConv2019 ; teja2018hydranets , and some further suggest sparsity regularization over experts via the gating mechanism shazeer2017outrageously ; wang2018deep . These gating mechanisms can be seen as a dynamic routing, which activates a single or a group of experts in the network on a per-example basis. Inspired by these works, our methods leverage PEA principled methods for automatically designing neural network inner components.

Furthermore, optimizers based on PEA theory may be useful for the neural architecture search phase. Common stochastic gradient-descent (SGD) and a set of PEA approaches, such as follow-the-regularized-leader (FTRL), were shown by 

shalev2012online ; hazan2016introduction ; van2014follow to be equivalent. Current NAS methods NASNET ; zoph2016neural ; ENAS ; liu2018darts ; cai2018proxylessnas ; wu2018fbnet ; li2019random ; hundt2019sharpdarts ; noy2019asap use Adam, SGD with Momentum or other common optimizers. One notion that is common in PEA principled methods is the regret cesa2006prediction . PEA strategies aim to guarantee a small regret under various conditions. We use the regret as a NAS objective, in order to establish a better principled optimizer than existing methods li2019random ; sciuto2019evaluating . Several gradient-descent based optimizers, such as Adam, present a regret bound analysis, however, the worst-case scenario for Adam has non-zero average regret reddi2019convergence , i.e., it is not robust. Our optimizer is designated for selecting architecture weights while achieving an optimal regret bound.

6 Conclusion

In this paper we presented XNAS, a PEA principled optimization method for differential neural architecture search. Inner network architecture weights that govern operations and connections, i.e. experts, are learned via exponentiated-gradient back-propagation update rule. XNAS optimization criterion is well suited for architecture-selection, since it minimizes the regret implied by sub-optimal selection of operations with tendency for sparsity, while enabling late bloomers experts to warm-up and take over during the search phase. Regret analysis suggests the use of multiple learning rates based on the amount of information carried by the backward gradient. A dynamic mechanism for wiping out weak experts is used, reducing the size of computational graph along the search phase, hence reducing the search time and increasing the final accuracy. XNAS shows strong performance on several image classification datasets, while being among the fastest existing NAS methods.


We would like to thank the members of the Alibaba Israel Machine Vision Lab (AIMVL), in particular to Avi Mitrani, Avi Ben-Cohen, Yonathan Aflalo and Matan Protter for their feedbacks and productive discussions.


7 Proofs

7.1 Proof of Lemma 1

Proof: By contradiction, assume that expert is being wiped-out at the iteration based on line 11 in algorithm 1, and without loss of generality, is the leading expert at that time,


Since expert is the optimal one in hindsight, specifically,


However, since the loss is -bounded, the ratios between weights at time are bounded as well,


Recap inequalities (8)-(10),

In contradiction to (7.1). ∎

7.2 Proof of Lemma 2

Proof: Left Hand Side. Due to the non-negativity of for all , we have,




Right Hand Side. Denote as the cardinally of the set . Then,


where (13) is since and (14) is since , according to line 12 of algorithm 1. Thus taking the telescopic product yields,


Finally we have,


7.3 Proof of XNAS Regret Bound

Proof: First let us state an auxiliary lemma,

Lemma 3 (Hoeffding).

For any random variable

with and ,


Proof can be found in hoeffding1953lower .

We start with defining experts’ auxiliary and accumulated-auxiliary losses following cesa2006prediction ,


Notice that the auxiliary losses are bounded by an input parameter in line 1 of algorithm 1,


We also define the set of remaining experts at time ,


Such that,


We Now bound the ratio of weights sums from both sides.
Let us derive a lower bound:


Where we used lemma 1 in (23), assuring that loss minimizer is among the remaining experts.
Let us derive an upper bound:


Where (24) is due to (22) and (25) is by setting,


Inequality (26) results from lemma 3, with and (27) is a result of the linearity of .

Summing the logs telescopic sum:


Setting with bounds specified in by lemma 2, we have,


Combining the lower and upper bounds and dividing by ,


We now bound the accumulated regret, using the convexity of the loss,


Combining 31 and 32 completes the proof,


8 Supporting Materials for the Key Properties Discussion

8.1 A Deterministic 3D Axes Toy Problem

In an attempt to demonstrate the possible recovery of late bloomers, we view an optimization problem in a three dimensional space as a prediction-with-experts problem, where each axis represents an expert with a constant prediction, i.e. for the axes respectively. The forecaster then makes a prediction according to the following,

with in the simplex, i.e. . Setting for , for a given loss function , the update terms for GD with softmax (section 3.2.1) and XNAS (section 2.2), associated with the axis, are as follows,

as the corresponding terms for the and axes are similar by symmetry, see a full derivation in section 8.4.2.

Now let us present the following toy problem: A three dimensional linear loss function is being optimized for gradient steps over the simplex, i.e. . Then for illustrating a region shift in terms of the best expert, the loss function shifts into another linear loss function which is then optimized for additional steps. The trajectories are presented in Figure 3.

Figure 3: Trajectories of XNAS and GD with softmax in 3D (left), projected on the simplex (middle) and the update terms for the axis (right). At the middle, color indicates progression through time, starting darker and ending lighter

At the first stage the axis suffers many penalties as its weight shrinks. Then it starts receiving rewards. For GD with softmax, those rewards are attenuated, as explained in section 3.2.1 and can be seen in Figure 3 (right). Despite the linear loss, with constant gradients, the update term decays. Note that this effect is even more severe when dealing with more common losses of higher curvature, where the gradients decay near the local minimum and then further attenuated, as illustrated in section 8.2.4. Once the gradient shifts, it is already depressed due to past penalties, hence the axis struggles to recover. XNAS, however, is agnostic to the order of the loss values in time. Once the rewards balance out the penalties, the path leads towards the axis. In the meanwhile, the axis takes the lead.

8.2 A Deterministic 2D Axes Toy Problem

In section 8.1 we show the built-in attenuation of weak operations by GD with softmax. This is illustrated by a three dimensional toy example where the axes represent experts of constant predictions. Here we elaborate on this effect using a similar two dimensional toy problem, where the gradients with respect to the axes are the negative values of one another. See section 8.4.3 for the setup and full derivations. All the experiments in this section are conducted using a learning rate of for both optimizers for steps.

8.2.1 Balanced Initialization

Figure 4: Trajectories of GD with softmax and XNAS for a (left) the distance from the optimal point (middle) and the corresponding magnitude of the gradients (right) for a balanced initialization . The vertical lines are the contour lines of the loss function. The dashed line is the simplex , and the trajectories are the solid lines with circles at their ends.

Figure 4 illustrates the attenuation of gradients for GD with softmax, as although the gradients of the loss are constant, the gradients’ magnitude decreases as we move away from the initialization , i.e. . XNAS indeed receives constant gradients thus reaches the minima faster.

8.2.2 Imbalanced Initialization

The attenuated gradients also make GD with softmax more sensitive to the initialization, as demonstrated in Figure 5, where and GD with softmax, whose gradients are attenuated, makes no progress while XNAS reaches the minima.

Figure 5: Trajectories of GD with softmax and XNAS for a (left) the distance from the optimal point (middle) and the corresponding magnitude of the gradients (right) for an imbalanced initialization . The vertical lines are the contour lines of the loss function. The dashed line is the simplex , and the trajectories are the solid lines with circles at their ends.

8.2.3 The Preference of Dense Solutions

Presenting the attenuation factor on the simplex, i.e. , in Figure 6, demonstrates how gradients are harder attenuated as far away as the variables move from a dense solution, e.g. at .

Figure 6: The attenuation factor of GD with softmax along the simplex (left) and versus as (right)

Hence, it is harder for GD with softmax to strengthen a single expert over the other. This effect encourages dense solutions over sparse solutions, i.e. a choice of a single expert. Due to the descretization stage, described in section 2.1, the denser the solution is, the more degradation in performance is incurred. Hence dense solutions should be discouraged rather than encouraged.

8.2.4 Loss Functions of a Higher Curvature

Sections 8.2.1 and 8.2.2 deal with a linear loss function of no curvature and constant gradients. Once convex loss functions are considered, the gradients decrease towards the local minimum. Figure 7 illustrates the further attenuation of GD with softmax for a quadratic loss function , which makes it even harder for GD with softmax to reach the local minimum.

Figure 7: Trajectories of GD with softmax and XNAS for a (left) the distance from the optimal point (middle) and the corresponding magnitude of the gradients (right). The circular lines are the contour lines of the loss function. The dashed line is the simplex , and the trajectories are the solid lines with circles at their ends.

8.3 Regret and Correct Selection in Statistical Setting

Figure 8: Correct selection fraction and regret of optimizers.

We consider a statistical setup for comparing XNAS with the common Gradient Descent (GD) with softmax, described in section 3.2.1. This setup simulates the iterative architecture optimization and final selection of the top expert for a single forecaster.

Two forecasters are compared, XNAS and GD with softmax. Both receive noisy independent and identically distributed (i.i.d) rewards of experts. Each expert has an initial i.i.d bias simulating its inherent value, so its rewards satisfy for and , where

is a Gaussian distribution with a mean

and a standard deviation


The first forecaster updates its weights using GD with softmax update rule from  Equation 6 (full derivation in section 8.4.1), common to previous NAS methods, while the second is using Algorithm 1.

The forecasters use their update rules to update weights along the run. At the end of the run each selects the expert with the largest weight. A correct classification satisfies . The average regret of those runs is also calculated based on equation 2.

Figure 8 shows a mean of Monte-Carlo runs, each of time-steps, plotting the regret and the fraction of correct selection (classification). In Figure 8 (left), both terms are plotted versus a varying number of experts. It can be seen that the regret of XNAS is significantly smaller, scaling with the number of experts like , as implied by its regret upper bound in equation 4, while GD regret scales like  hazan2016introduction .

In Figure 8 (right), the noise standard deviation is varying, making it harder to correctly classify the expert with the highest bias. Again, XNAS dominates GD with softmax, which is more sensitive to the noisy rewards due to the ’late bloomers’ described in 3.2.1, e.g. the best experts might suffer some large penalties right at the beginning due to the noise, thus might not recover for GD with softmax.
In both graphs it can be seen that the correct selection fraction is monotonically decreasing as the regret is increasing. This gives an additional motivation for the use of the regret minimization approach as a criterion for neural architecture search.

8.4 Gradients Derivations

For the comparison to previous work liu2018darts , we consider the decision variables , as the right hand side is defined at (1).

8.4.1 The Derivation of Derivatives in the General Case


where is the Kronecker delta.

Observe that: