Deep learning has been successfully applied in many fields including image recognition, speech recognition [Hinton et al.2012], and machine translation [Sutskever et al.2014, Yonghui et al.2016]. To achieve good performance, these deep architectures need to be carefully designed by human experts. Due to the huge search space, it definitely takes great efforts to do this hard job. In recent years, there has been an increase of the literatures about neural architecture search [Liu et al.2018a, Liu et al.2017, Kandasamy et al.2018]
. Roughly, such works can be categorized into two main streams, one is based on reinforcement learning (RL)[Zoph and Le2016]
, and the other is based on evolutionary algorithm (EA)[Liu et al.2017, Real et al.2018]. EA-based methods can continually update the structures of neural nets to generate a series of generations for expanding search space through simpler and more efficient operations, and show promising results compared with other methods including RL-based ones. However, EA-based methods need to evaluate a large number of individual networks, which consumes a lot of expensive GPU resources.
To solve this problem, one can use Bayesian optimization strategy. The conventional solution of automatic machine learning resorts to formalizing machine learning process as a black-box optimization task. Bayesian optimization (BO) is an effective global optimization algorithm, with the goal of finding the optima of black-box objectives. Since the information about objective function is not known, BO utilizes a surrogate model to fit the black-box function, actively selects the most potential samples for real-world evaluations based on fitting results, and can quickly get the optimal location of the objective with only a few attempts. BO takes advantage of the information from the previous evaluations to update the quality of the surrogate model, which in turn, will help acquisition function to make better decision about where to evaluate next time. This is the main rationale why BO works more efficient. BO has a large number of applications including hyperparameter tuning for machine learning models[Feurer et al.2015, Thornton et al.2013].
BO can be applied to improve the EA-based neural architecture search methods. By designing a good surrogate function, which can appropriately fit the classification or regression accuracy of a given neural architecture, BO can help EA-based method find an approximately optimal architecture by evaluating only a few candidates, and thus greatly reducing the computation overhead and saving GPU time. The current methods use Gaussian processes (GPs) as surrogate function, define the similarity between two neural architecture according to their structure difference, and then construct a similarity matrix as the kernel matrix of GPs [Kandasamy et al.2018, Jin et al.2018]. Because the function mapping the structure of a neural architecture to its predictive performance is black box, it is difficult to know which features are appropriate or even relevant. The features that affect prediction may be structural features, attributes of nodes, global features of networks, or even some implicit features. Therefore, it is difficult for the man-made similarity matrix to reflect the characteristics of neural architecture in an all-round way, which maybe responsible for the failure of finding optimal solution. In addition, these manually customized kernel matrices contain subjective factors, and different definitions often lead to different results. Moreover, GPs is not scalable. Both the training and prediction of GPs involve computing the inverse of kernel matrix, with a time complexity of . When the number of observation samples becomes large, training and prediction become very slow, making it impossible to explore a larger search space.
In view of these problems, we plan to propose a new surrogate that can automatically extract useful features from neural architectures, and these features can be used to fit the mapping from architecture to predictive performance. Specifically, we model neural networks as attributed graphs and model the task of neural architecture search as the task of attributed graph optimization. Instead of GPs, we use Bayesian graph neural network (GNN) as new surrogate. GNN is a deep model of graph representation learning, which supports supervised learning of node and link embeddings from the context of attributed graphs. Its parameter sharing mechanism can greatly drop model complexity, which can not only reduce the time of training and prediction, but also avoid over-fitting. These advantages of GNN are especially suitable for the BO in which less training data are available. After embedding each layer, the representation of overall neural architecture can be obtained by pooling operations.
Based on the new surrogate, we develop a graph Bayesian optimization framework to address the problem of attributed graph optimization. We name it NASGBO, i.e., Neural Architecture Search with Graph Bayesian Optimization. Figure 1 illustrates its workflow. The proposed surrogate is given at the top. It consists of a GNN layer, a pooling layer, a MLP layer and a BLR layer. The input of GNN is the attributed graph encoding an input neural architecture, and its output is the embedding of nodes and links. Pooling layer combines node/link embeddings into the representation of the entire attributed graph and outputs it to MLP layer. MLP layer includes multiple fully connected layers, and predicts the predictive accuracy of the input neural architecture according to its embedding. In order to capture the uncertainty of MLP prediction, we add a BLR (Bayesian linear regression) layer. Note that we only add uncertainty at the last layer of the surrogate model, rather than modeling all model parameters as random variables. This is to balance the need of uncertainty measuring and the cost of computation.
By randomly generating, training and testing some neural architectures, we first prepare an initial training set. The training set is used to train the surrogate model (step a). Based on the current training set, a new population is generated by evolutionary operations (step b). The surrogate is used to predict the performance of each new individual, and one potential individual is selected out of them by maximizing the acquisition function (step c). Then the individual is trained, tested, and added to the current training set (step d). Under some cost constraints, the process is repeated until an approximate optimal solution is returned. Each component of the framework will be elaborated in remaining text.
2 Problem Statement
At present, there are two main ways to describe the architecture of neural networks [Elsken et al.2018]. One is multi-branch chained architecture [Kandasamy et al.2018] and the other is cell-based architecture [Liu et al.2018b, Liu et al.2018c]. In the former representation, each layer can choose different operations such as pooling, convolution, etc. The output of each layer can be used as the input of all subsequent layers, not just as the input of the next layer. Resnets and Densenets belong to this type. This way is more flexible and can generate any kinds of network architectures, so the search space is correspondingly very large. In the latter representation, a neural network is constructed by cells as building blocks, e.g., normal cell and reduction cell. Because the structures of cells are fixed, one only needs to optimize the graphs of cells, so cell-based representation can greatly reduce the search space and allow us to generate deeper network architectures. Limited by space, this work only focus on the multi-branch chained architecture as an example to demonstrate and verify our framework because it has larger search space with more challenges. It is worth noting that our proposed framework can be readily applied to the cell-based architecture by concentrating cells into nodes.
2.1 Objective Formula
Let be the performance evaluation function of neural architecture, where denotes search space and denotes predictive accuracy. The goal of neural architecture search is to find the architecture with the highest predictive accuracy, so the objective function can be formalized as:
Each architecture in search space is modeled as an attributed graph , where is a set of nodes denoting the layers of neural architecture, is a set of edges, is the feature set of nodes and is the global feature set of the graph.
2.2 Design of Search Space
One of the key points is how to construct the neural network architecture search space . The search space defines the variables of the optimization problem and is also different corresponding to different search algorithms. In order to apply the GNNs method to Bayesian optimization, we use attributed graph to represent each neural network architecture. We encode the neural network into an attributed graph as input, which can fully exploit various features of the architecture. The attributed graph is mainly composed of network structure, layer attributes, and global attributes of the architecture. The network structure consists of a layer set and a directed edge . The directed edge indicates whether the output of layer node is the input of the next layer . Each node here is treated as a layer.
Taking CNNs as an example, each layer is a node. Node attributes include layer types (softmax, conv3, conv5, conv7, res3, res5, res7, fc, max-pool, avg-pool, ip, op, etc), which each layer type occupys a single dimension in the node features. Other node attributes include the number of convolution units, the number of fully connected units, etc. The global graph feature include the num of the various node types, the num of nodes, the num of edges, etc. The connected edges between nodes represent the output of upper layer as the input of next layer. The index number of the node is obtained by an topological ordering, that is to say, a node with a smaller index number always points to a node with a larger index number.
Figure 2 provides an illustration from a CNN to the corresponding attributed graph. there are 8 nodes in the figure. To simplify the description, only first few dimensions of node attributes have been shown here, i.e. ip, op, cov3, pooling, fc, softmax, op and the number of units. For example, the attributes of node 2 include cov3 and the number 16 of units.
-point bold type. Leave a blank space
3 The components of NASGBO framework
As illustrated by the Figure 1, our proposed NASGBO framework consists of three main components: deep surrogate model, acquisition function and evolutionary strategy. The workflow of NASGBO is described in Algorithm 1.
3.1 Bayesian GNN surrogate
Due to the issues of the GPs as mentioned in the introduction, we use a deep graph neural network as the surrogate function rather than GPs. In addition, to make our surrogate more scalable and be able to model uncertainty we integrate a layer of BLR (Bayesian linear regressor).
Graph Neural Networks (GNNs) is a deep learning method designed for graph data [Battaglia et al.2018]
, which can be used to learn the representation of an attributed graph. GNN generalizes various neural network methods for manipulating graphs and defines a class of functions for relational reasoning based on graph structure representations. GN (graph network) module is the main computational unit of the GNN, which is a “graph to graph” module that takes the graph as the input and returns a graph as its output. The nodes and edges have their own attributes, which can be a vector or tensor.
In our framework, a GN block that we adopt contains three “update” functions:
where , , , , and have five layers, is an element-wise sum operation, represents a vector of the global attributes, is a set of nodes, each represents the attributes of the node , is a set of edges, where each represents the attribute of the edge , is the index of the receiving node, and is the index of the sending node, , and . The edge attribute is updated with the node attribute connected to the edge, the node is updated with the edge attribute connected to the node.
Using the pooling layer, we make full use of the node features, the edge information of the graph and the global features to learn the global representation of the whole graph.In order to capture the uncertainty of the graph when predicting the metric of the graph, we add a Bayesian linear regression (BLR) as the last layer of the surrogate architecture behind multiple fully connected layers. We refer to this model as adaptive basis regression, which are parameterized by the weights and biases of the deep neural network. The form of the BLR is as follows:
where y is the output of the surrogate function and
,which is normal distribution, andis the decision matrix output by previous layers as the input of BLR layer. Given a prior distribution on weights: , where denotes the uncertainty of .
The measure of can be predicted by:
where are observations, are evaluated measures,
For Bayesian optimization, if surrogate function is the GPs, Maximizing the acquisition function takes time to calculate the inverse of surrogate matrix .For Deep Graph Bayesian Optimization, it takes and time to maximize the acquisition function. M is the number of units on BLR layer(usually ). Therefore, the Deep Graph Bayesian Optimization takes less time when maximizing the acquisition function.
3.2 Acquisition function
In Bayesian optimization, it can be seen that the probability description of objective function
can be quantified by sampling. Usually the acquisition function mainly uses exploiting and exploring sampling ideas. Exploring new spaces helps to estimate a more accurate objective function, and sampling near the existing results (usually near the maximum) are expected to find a larger one. The purpose of the acquisition function is to balance the two sampling ideas and quantify the potential of candidate graphs based on previous validations. Given a graph search space and a hyper-parameter space , we can define the acquisition function . Although any other acquisition functions can be used in the proposed framework, this paper uses expected improvement (EI), a simple, valid, and common criterion, as the acquisition function. The EI function is the expectation of the improvement function at candidate point . Specifically, it can be fulfilled by
3.3 Evolutionary strategy
Unlike NASBOT and NASNM, We firstly use an evolutionary algorithm (EA) to generate neural networks of generations rather than optimising the acquisition function . The value of may be 1, 2, etc. Then we use deep graph bayesian optimization to select the most potential neural network . However , by using EA to optimise the acquisition function , the selected neural network may be very deep and lead to resource crunch after many generations of mutation. This is the main reason why we use EA only for generating. The above is just a simple strategy to balance the breadth and depth of mutation. It can also be tackled in other good strategies.
Each neural network is a feasible solution in the search space, and an approximate evaluation is needed to judge the performance of neural network. We encode each neural network model into a graph. The mutation operation of a neural network model can be transformed into operation on the graph, such as adding a node, deleting a node, adding an edge, etc. For example, adding a convolution layer is equivalent to adding a node to the graph. The training environment and mutation operations of NASGBO are the same as NASBOT. Table 1 shows several types of operations.
|Adding skip||Randomly select two convolution layer ids for skip connection|
|Incresing units||Increase the number of units by 1/8|
|Decresing units||Decrease the number of units by 1/8|
|Adding layer||Randomly select two convolution layer ids for adding a convolution layer|
|Removing layer||Randomly select a convolution layer id for removing|
4 Experiments and Analysis
To evaluate the performance of the proposed Neural Architecture Search with Graph Bayesian Optimization (NASGBO), we compare it with the following baselines. We mainly verify the performance of NASGBO from two perspectives including accuracy and cost.
4.1 Baseline Algorithms
RAND: In the case of the same initial individuals, rand algorithm selects one from initial individuals for mutation and evaluation.
TreeBO [Jenatton et al.2017]: A BO method that only searches over feedforward structures.
NASNM [Jin et al.2018]: A Bayesian optimization algorithm in which the network morphism is used to construct kernel function.
NASBOT [Kandasamy et al.2018]:A Bayesian optimization algorithm for neural architecture search. Firstly, NASBOT utilizes Optimal Transport Metrics for Architectures of Neural Networks to represent the similarity of the networks, and then searches neural network architectures through the GPs.
Unlike NASBOT and NASNM, we use graph neural network (GNNs) as the surrogate function. It can adequately represent the architectural features of the neural networks.
We use five data sets for the experiments, as follows: Indoor Location [Torres-Sospedra et al.2014], Slice Localisation [Graf et al.2011], Cifar10 [Krizhevsky and Hinton2009], Minist [LeCun et al.1998], Fanshion Minist [Xiao et al.2017]
. The first two data sets are applied to the regression problem of MLPs. The last three data sets are applied to the classification tasks of CNN images. For the first two data sets, we split the data sets using a scale of 0.6-0.2-0.2, which used as training data sets, test data sets, and validation data sets respectively, and normalize these data sets to have zero mean and unit variance. For the Cifar10 data sets, there are 60,000 images, of which 50,000 are for training and 10,000 are for testing. We used a 40K-10K-10K ratio to segment the data sets, which used as training dataset, test dataset and validation dataset. For the last two data sets, there are 70,000 images, of which 60,000 are for training and 10,000 are for testing. We used a 50K-10K-10K ratio to segment the data sets.
Our method is executed on a single GPU (NVIDIA GeForce GTX 1080 Ti). The regression MSE or classification error (lower is better) on the test set is selected as the evaluation metric.
After running for a period of time (such as 12 hours), the corresponding accuracy of neural network may not increase in the next period of time. Finally, we train the best neural network on the test set to get results of test set.
Table 2 shows that the results on the test set with the best model. As shown by Table 2, NASBGO can get the best result of test set on Cifar10, Fashion, Mnist, and Slice data sets. NASBOT can get the best result on Mnist and Indoor data sets. Relatively speaking, NASBGO performs better than other algorithms in getting the best network architecture.
We use the number of evaluations as cost metric, which does not contain the number of initialized neural networks. The training environment and mutation operations of NASGBO is consistent with NASBOT. We implemented NASBOT conscientiously, and use it as comparative experiment against the number of evaluations on the five data sets .Rand and EA algorithm can be implemented easily compared to our algorithm. We also add them to the comparative experiment.
Figure 3 shows best validation score for each method against number of evaluations for CNNs. As shown by the Figure 3, NASGBO converges faster than NASBOT on Cifar10 and Fashion data sets in terms of the number of evaluations. The performance of two algorithms is almost the same on Minist data sets.
Figure 4 shows best validation score for each method against number of evaluations for MLPs. As shown by the Figure 4, NASGBO converges faster than NASBOT on slice. For the Indoor data set, NASBOT is slightly better than the algorithm we proposed.
For CNNs and MLPs, we analyze the results of optimal architectures we obtained through fashion dataset and slice dataset. Figure 5 shows optimal architectures of NASNM, NASBOT and NASGBO on fashion dataset and Figure 6 shows optimal architectures of NASNM, NASBOT and NASGBO on slice dataset.
As shown by the Figure 5, NASNM uses 5 residual blocks, NASBOT does not use any residual block, and NASGBO uses two residual blocks. From the overall perspective, we get a simpler architecture. According to table 2, we can see that classification error on the test set is lower.
For the slice dataset, the initialized architectures are straight-chain without skip connections. From the results, main reasons for affecting the results of slice dataset are the size of architecture and the skip connection of architecture. As shown by the Figure 6, NASNM can learn about the impact of skip connections on the architecture, and add many skip connections. However, as the size of the architecture grows, the results get worse. NASBOT can also learn a simpler architecture than NASNM, but it is still relatively complicated compared to the architecture we have learned and the value of RMSE is higher than ours. With the use of skip connections, our algorithm makes the architecture as simple as possible. Finally, we can get better result on slice dataset from table 2.
Overall, our method can achieve competitive accuracy with less overhead for on MLPs and CNNs, the two benchmark neural architecture search tasks.
The main contributions of this work are two-fold. (1) We model neural networks as attributed graphs and model the task of neural architecture search as the task of attributed graph optimization. Based on this idea, we develop a graph Bayesian optimization framework to address neural architecture search, which integrating the advantages of EA-based methods and BO-based methods by designing a new surrogate model based on Bayesian GNN. This new surrogate can automatically extract features from deep neural architectures, and can use such features to fit and characterize the black-box mapping from architecture to prediction as well as its uncertainty. Moreover, the training and prediction time of Bayesian optimization can be reduced to linear time by using the Bayesian GNN surrogate, instead of the cubic time of Gauss process, which make our method be able to explore much larger search space and find better solutions. (2) We rigorously show that our method outperforms the state-of-the-art works on benchmark neural architecture search tasks.
- [Battaglia et al.2018] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.
- [Elsken et al.2017] Thomas Elsken, Jan-Hendrik Metzen, and Frank Hutter. Simple and efficient architecture search for convolutional neural networks. arXiv preprint arXiv:1711.04528, 2017.
- [Elsken et al.2018] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. arXiv preprint arXiv:1808.05377, 2018.
- [Feurer et al.2015] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems, pages 2962–2970, 2015.
- [Graf et al.2011] Franz Graf, Hans-Peter Kriegel, Matthias Schubert, Sebastian Pölsterl, and Alexander Cavallaro. 2d image registration in ct images using radial image descriptors. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 607–614. Springer, 2011.
- [Hinton et al.2012] Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6):82–97, 2012.
- [Jenatton et al.2017] Rodolphe Jenatton, Cedric Archambeau, Javier González, and Matthias Seeger. Bayesian optimization with tree-structured dependencies. In International Conference on Machine Learning, pages 1655–1664, 2017.
- [Jin et al.2018] Haifeng Jin, Qingquan Song, and Xia Hu. Efficient neural architecture search with network morphism. arXiv preprint arXiv:1806.10282, 2018.
- [Kandasamy et al.2018] Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with bayesian optimisation and optimal transport. In Advances in Neural Information Processing Systems, pages 2020–2029, 2018.
- [Krizhevsky and Hinton2009] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
- [LeCun et al.1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
- [Liu et al.2017] Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hierarchical representations for efficient architecture search. arXiv preprint arXiv:1711.00436, 2017.
[Liu et al.2018a]
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li,
Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy.
Progressive neural architecture search.
Proceedings of the European Conference on Computer Vision (ECCV), pages 19–34, 2018.
- [Liu et al.2018b] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European Conference on Computer Vision (ECCV), pages 19–34, 2018.
- [Liu et al.2018c] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018.
- [Real et al.2018] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01548, 2018.
- [Sutskever et al.2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112, 2014.
- [Thornton et al.2013] Chris Thornton, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 847–855. ACM, 2013.
- [Torres-Sospedra et al.2014] Joaquín Torres-Sospedra, Raúl Montoliu, Adolfo Martínez-Usó, Joan P Avariento, Tomás J Arnau, Mauri Benedito-Bordonau, and Joaquín Huerta. Ujiindoorloc: A new multi-building and multi-floor database for wlan fingerprint-based indoor localization problems. In Indoor Positioning and Indoor Navigation (IPIN), 2014 International Conference on, pages 261–270. IEEE, 2014.
- [Xiao et al.2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
- [Yonghui et al.2016] W Yonghui, M Schuster, Z Chen, QV Le, M Norouzi, W Macherey, M Krikun, Y Cao, Q Gao, K Macherey, et al. Bridging the gap between human and machine translation. arxiv preprint. arXiv preprint arXiv:1609.08144, 2016.
- [Zoph and Le2016] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016.