Standard centralized machine learning methods require to collect training data from distributed users and stored them on a single server, which suffers from a high risk of leaking users’ private information. Therefore, a distributed approach called federated learning  was proposed to preserve data privacy, enabling multiple local devices to collaboratively train a shared global model while the training data remain to be deployed on the edge devices. Consequently, the central server has no access to the private raw data and the client privacy is protected.
However, federated learning demands a large amount of communication resources in contrast to the conventional centralized learning paradigm, since updating the global model needs to frequently download and upload model parameters between the server and edge clients. To mitigate this problem, a large body of research work has been carried out to reduce the communication costs in federated learning. The most popular approaches include compression and sub-sampling of the client uploads [40, 24, 7] or quantization of the weights of the models . Most recently, Chen et al.  suggest a layer-wise parameter update algorithm to reduce the number of parameters to be transmitted between the server and the clients. In addition, Zhu et al. 
uses a multi-objective evolutionary algorithm (MOEA) to simultaneously enhance the model performance and communication efficiency.
Little work has been reported on offline optimization of the architecture of deep neural networks (DNNs) for federated learning, let alone real-time neural architecture search (NAS) suited for the federated environment. In the field of centralized machine learning, Zoph et al. 
present some early work on NAS using reinforcement learning, which, however, consumes a plenty of computational resources. To mitigate this problem, Phamet al.  introduce a directed acyclic graph (DAG) based neural architecture representation to significantly accelerate the search speed without much degradation of the learning performance by using weight sharing technique.
Recently, evolutionary approaches to NAS have received increasing attention [41, 35, 36, 29], and hybrid methods that combine evolutionary search with the gradient method  have also been reported . To reduce the computational cost, surrogate ensembles have been introduced in evolutionary NAS, which has been shown promising in reducing the computational complexity without deteriorating the learning performance .
Most existing search strategies for NAS in a centralized learning environment are not well suited for federated NAS for the following reasons. First, most current search strategies for centralized learning focus on improving the model performance without paying much attention to the model size and computation cost. Client devices like mobile phones cannot afford computationally intensive model training and bandwidth restrictions do not allow very large models to be transmitted frequently between the server and clients. Second, many NAS techniques for accelerating model training [34, 35, 29, 13, 30, 46]
adopt transfer learning techniques[32, 47]
by searching upon cell based small models and transferring the found promising cell structures to large models. Such techniques are not directly applicable to federated learning, because such cell transfer methods may cause model divergence in a distributed training scheme and learning transferred models from scratch will consume extra communication resources. Finally, modern deep neural networks may fail to work in federated learning as learnable parameters in the batch normalization layer
may degrade the global model performance after model aggregation, because the locally trained models may have very different weights in mean and variance.
Note that our previous work on multi-objective evolutionary federated optimization  is an offline evolutionary approach to NAS, like most conventional evolutionary optimization algorithms. In offline evolutionary NAS, the parameters of a newly generated offspring model are randomly reinitialized and trained from scratch before the model is evaluated on a validation dataset, requiring a large amount of computational resources. What is worse, the performance of the reinitialized models will dramatically degrade, making the offline optimization approach infeasible for realtime application of federated learning, such as online recommendation systems .
Therefore, offline federated evolutionary optimization of neural networks is not applicable to the real-world applications and it is highly desirable to develop a framework for real-time federated evolutionary neural architecture search. The main contributions of this work are summarized as follows:
A double-sampling technique is proposed that randomly samples a sub-network of the global model, whose parameters are transmitted to a few randomly sampled local devices without replacement. The number of devices to be sampled depends on the ratio between the number of connected local clients and the number of individuals in the population. The double-sampling technique brings about two advantages. First, only a sub-network needs to be trained on local devices, significantly reducing the number of parameters to be uploaded from the local devices to the server. Second, sampling of the clients without replacement makes sure that each local device needs to train only one sub-network for once at each generation. The above two features together make the proposed real-time evolutionary method substantially different from offline evolutionary NAS, where the whole neural network must be trained on all local devices for fitness evaluations, and each device needs to train all networks in the population. To the best of our knowledge, it is the first time that a real-time evolutionary NAS algorithm has been developed for the federated learning framework.
An aggregation strategy is developed that updates the global model based on the sub-networks sampled and trained at each generation. Thus, in the proposed real-time evolutionary NAS, each generation is equivalent to a training round in the traditional federated learning. In addition, the weights of the sampled sub-networks are inherited before it is trained on a local device, accelerating the convergence and avoiding dramatic performance deterioration caused by random reinitialization.
Extensive comparative studies are performed to verify the performance of the proposed real-time federated evolutionary NAS by comparing the learning performance and computational complexity of the models it obtains with that of the standard ResNet18  on both IID and non-IID data. We also show the proposed real-time evolutionary NAS is at least five times faster than the offline evolutionary NAS method.
In this section, a review of federated learning is given at first, followed by an introduction to NAS for deep neural networks. Then, the basic of a multi-objective evolutionary algorithms will be introduced. Finally, a brief discussion is given to clarify the differences between the offline and real-time evolutionary optimization frameworks.
Ii-a Federated Learning
As mentioned before, federated learning is an emerging decentralized privacy-preserving model training technology that enables local users to training a global model without uploading their private local data to a central server. A conventional federated learning algorithm called federated averaging (FedAvg) algorithm is shown in Algorithm 1. In the following, we briefly introduce this algorithm.
Ii-A1 Server Side
The model parameters are initialized once at the beginning of FedAvg algorithm, which is then sent to all selected clients, where is the number of total clients and is the fraction of all clients between and . After all clients update and send the updated local model parameters back to the central server, the parameters of the global server model will be replaced by the weighted averaging of each client’s model parameters .
Ii-A2 Client Side
The local model parameters are replaced by the downloaded global model parameters
. Then the local model parameters are updated by the batch stochastic gradient descent (SGD) algorithm, where is the local learning batch size. After local training, the learned model parameters will be sent back to the central server for global model aggregation.
An alternative approach to local execution is to calculate and upload the local model gradients only. Then, they are aggregated on the server by , where represents local gradients. This method is beneficial to reduce the local computation consumption while sharing the same computing result with the previous one.
Ii-B Neural Architecture Search
has been extremely successful in the field of image recognition and speech recognition. However, most deep neural network (DNN) models are manually designed by human experts, and not until very recently has increasing attention been attracted to automatically search for a good model architecture using NAS methods. The search space in NAS depends on the neural network model in use and in this work, we limit our discussions to convolutional neural networks (CNNs)[28, 26].
Roughly speaking, the search space for NAS can be categorized into macro and micro search spaces . The macro search space is over the entire CNN model, as shown in Fig. 1. The whole model consists of sequential layers where the dashed lines are shortcut connections, similar to ResNet . So the macro search space aims to represent a good structured model in terms of the number of hidden layers , operations types (e.g. convolution), model hyper parameters (e.g. convolutional kernel size), and the link methods for shortcut connections. Different from the macro search space, micro search space only covers repeated motifs or cells [45, 19] in the whole model structure. And these cells are built in complex multi-branch operations [43, 44] as shown in Fig. 2, where the given structure contains two inputs and coming from two previous layers and only one concatenated output.
. RL-based search methods always adopt a recurrent neural network (RNN) as a controller to sample a new candidate model to be trained and then use the model performance as the reward score. And then this score can be used to update the controller for sampling a better child model in the next iteration.
Apart from RL, evolutionary algorithms (EAs) based approaches are often used to deal with multi-objective optimization problems in NAS. Different from the previous neuro-evolution techniques [2, 49] that aim to optimize both the weights and architecture of neural networks, EA based NAS only optimizes the model architecture itself, and the model parameters are trained using conventional gradient descent methods [36, 41, 29, 35]
. Evolutionary NAS starts with randomly generating a population of parent models with different structures and train them in parallel. When all the models are learned for some pre-defined epochs, fitness values are calculated by evaluating the models on the validation dataset. After that, genetic operators such as crossover and mutation are applied on the parents to generate the offspring population consisting of new models. Then, a selection operation will be performed that selects the better offspring models to be the parents of the next generation, which is often known as survival of the fittest. This selection and reproduction process repeats for a certain number of generations until certain conditions are satisfied. Note that conventional evolutionary NAS needs to evaluate a population of neural networks at each generation and all newly generated neural models are trained from scratch, which are not suited for the online optimization task.
The gradient method became increasingly popular recently, mainly because its search speed is much faster than RL-based and evolutionary NAS methods. In [30, 13] relaxation tricks  are used to make a weighted sum of candidate operations differentiable so that the gradient descent can be directly employed upon these weights . No reinitialization of the model parameters is needed in this approach.
However, the gradient based technique requires much more computation memories than other approaches, since the overall network need to be jointly optimized. To fix this issue, a sampling strategy is proposed by [14, 4], in which only one or two paths (a path is a sub-network connecting the inputs and outputs) are sampled from a complex neural network for training. In addition, a weight sharing technique  is also suggested to avoid reinitialization of the newly sampled models, which saves a plenty of training time. Other techniques such as Bayesian optimization  is also a good approach reducing the computational complexity of NAS .
Ii-C Multi-Objective Evolutionary Optimization
Federated NAS is naturally a multi-objective optimization problem. For instance, the model performance should be maximized, while the payload transferred between the server and clients should be minimized. In the machine learning community, multiple objectives are usually aggregated to be a scalar objective function using hyperparameters. By contrast, the Pareto approach to solving multi-objective optimization problems has been very popular and successful in evolutionary computation, which has also been extended to machine learning . The main difference between the Pareto approach and the conventional aggregated approach to machine learning is that in the Pareto approach, no hyperparameters need to be defined and a set of models presenting trade-off relationships between the objectives will be obtained. Finally, one or multiple models can be chosen from the trade-off solutions based on the user’s preferences.
The elitist non-dominated sorting genetic algorithm, NSGA-II in short, is a very popular multi-objective evolutionary algorithm (MOEA) based on the dominance relationship between the individuals. The overall framework of NSGA-II is summarized in Algorithm 2.
The main idea of non-dominated sorting is to generate a set of ordered fronts based on the Pareto dominance relationships between the objective values of and solutions located in the same front cannot dominate with each other. Solutions in the first non-dominated fronts will have a higher priority to be selected. This sorting algorithm has a computation complexity of , where is the number of objectives and is the population size. To promote the solution diversity, a crowding distance that measures the distance between two neighboring solutions in the same front is calculated and those having a large crowding distance have a higher priority to be selected. The computation complexity of crowding distance calculation is in the worst case, when all the solutions are located in one non-dominated front. Readers are referred to  for more details.
Ii-D Online and Offline Evolutionary NAS
As previously discussed, existing evolutionary NAS methods, also including most RL and gradient based NAS algorithms, are meant for offline model optimization. In other words, they are not suited for scenarios in which neural architecture search must be performed while the network is already in use, such as in recommender systems or vision surveillance systems.
Conventional offline evolutionary NAS is not suited for real-time applications for the following reasons. First, a neural network model is not allowed to be randomly initialized when it is already in use, because random initialization will seriously deteriorate the performance of the neural network. Second, the available computational resource on client devices is limited and therefore, NAS should not require substantially more computational resource. However, EAs are population based search, and at each generation (i.e., time instant), a set of models (depending on the population size) must be assessed, which will considerably increase the computational costs of the clients. Moreover, NAS should not significantly increase the communication costs. This is again challenging for evolutionary NAS, where the parameters of multiple models (depending on the population size) must be transferred between the server and the clients. Finally, the models in offline evolutionary NAS are usually not fully trained to reduce computation time. Thus, they must be trained again at the end of the evolutionary search, which will incur additional communication costs.
Iii Proposed Algorithm
In this section, we will introduce the encoding method and model structure used in federated NAS at first. And then the two objectives to be optimized and the double-sampling method for evaluating the objectives are explained. Finally, the overall framework of the proposed evolutionary federated NAS is presented.
Iii-a Structure Encoding for Federated NAS
For real-time NAS, we adopt a light-weighted CNN as the master model, since communication cost is always a primary concern in federated learning. In addition, the search space should not be too large and the total number of layers should be limited to make it appropriate for real-time federated optimization.
The master model used in the proposed online federated NAS is shown in Fig. 3, where a convolutional block, choice blocks (each branch of choice block contains two convolutional or more advanced depthwise convolutional layers except for the identity block) and a fully connected layer are linked to build a DNN containing a maximum of hidden layers. Specifically, the convolutional block consists of three sequentially connected layers with a convolutional layer, a batch normalization layer 
, and a rectified linear (ReLu) layer. And one choice block is composed of four pre-defined branches of candidate blocks, namely identity block, residual block, inverted residual block and depthwise separable block, as shown in Fig.4. Thus, there are in total possible one path sub-networks. In addition, these four candidate blocks are categorized into two groups, one is called normal block whose input and output share the same channel dimension, whereas the other is called reduction block whose output channel dimension is doubled and the spatial dimension is quartered compared to its input.
Identity block links its input to its output directly (Fig. 4
(a)), which can be seen as a ’layer removal’ operation. For the structure reduction part, it just operates two branches of point-wise convolution with a stride ofat first and then concatenates these two outputs from the channel dimension. As a result, the spatial dimensions of the inputs are quartered and the filter channels are doubled through this identity reduction block.
Residual block contains two sequentially connected convolution blocks as shown in Fig. 4(b), where the normal block has the same structure as the residual block used in ResNet . Note that the reduction residual block does not contain shortcut connections while the normal block has.
expanding point-wise convolution layer, followed by a batch normalization layer and a ReLu activation function. 2) adepthwise convolution layer [9, 18], followed by a batch normalization layer and a ReLu activation function. 3) a spatially filtered point-wise convolution layer followed by a batch normalization layer.
The intuition of using expanding at the first layer instead of in the last layer as done in the bottleneck layer is that a nonlinear activation function such as ReLu may cause layer information loss 
and using a nonlinear projection upon a high-dimensional space can mitigate this issue. After the tensor is mapped back to a low-dimensional space again through the last point-wise layer, the ReLu function is not needed to prevent information loss.
Depthwise separable block consists of two depthwise convolution operations  (Fig. 4(d)) that incurs lower computation cost than the conventional convolution operation. It has been proved in  that depthwise convolution consumes about one-eighth to one-ninth of the computation time of the standard convolution at the expense of a small deterioration in performance.
For each communication round, only one branch of all choice blocks is sampled from the master model and then downloaded to a client device for reducing communication costs and computation resources required at the local device. This sampled sub-model can be encoded into a two-bit binary string with a total length of bits. Every two bits in the code represent one specific branch in the choice block. For instance, represents branch , which is the identity block, represents branch , i.e., the residual block, represents branch , the inverted residual block, and represents branch , which is the depthwise separable block. Therefore, the binary string (also called the choice key) can be decoded into a sub-model with structure as shown in Fig. 5.
Iii-B A Double-Sampling for Objective Evaluations
Offline evolutionary optimization is intrinsically not suited for federated learning. Although one or multiple light weighted models with high performance can be found by an offline evolutionary algorithm in the last generation , a large amount of extra computation and communication resources is required. For instance, for an EA of a population size of , each client in an offline evolutionary NAS algorithm must evaluate the fitness of individuals at each generation, which is times in extra compared to the gradient method. In addition, the models are repeatedly randomly reinitialized and trained from scratch on the clients, which may not be good enough for use during the optimization.
In order to address the above issues, a double-sampling technique is proposed here to develop a real-time evolutionary federated NAS, in which at each generation, the global model for each individual are sub-sampled from a common master model, whereas the clients for training the global model of one individual is sub-sampled from the participating clients. Specifically, a choice key is generated for each individual to randomly sample a sub-network from the master model to be this individual’s global model. Then, the global model of this individual (a sampled sub-model) will be downloaded to a randomly sampled subset of the participating clients. The number of clients to be chosen, say , for training one global model of an individual is determined by the ratio between the number of individuals and the number of participating clients, i.e., , where is the number of participating clients, is the total number of clients and is the participation ratio at the current round. Here, we assume that the number of clients is equal to or larger than that of the population size.
In this work, two objectives are to be optimized: one is the test error of each global model and the other is the floating point operations per second (FLOPs) of the model. Recall that the global model of each individual is a sub-model of the master model sampled using a choice key. Population initialization of the proposed methods is composed of the following four main steps.
Initialize the mater model. Generate the initial parent population containing individuals, each representing a sub-model sampled from the master model using a choice key. Sample clients for each individual without replacement. That is, each client should be sampled only once.
Download the sub-model of each parent individual to the selected clients and train it using the data on the clients. Once the training is completed, upload the local sub-models to the server for aggregation to update the master model.
Generate offspring individuals using crossover and mutation. Similarly, generate a choice key for each offspring individual to sample a sub-model from the master model. Download the sub-model of each offspring individual to randomly sampled participating clients and train it on these clients. Note that the weights of the sub-models are inherited directly from the master model and will not be reinitialized in training. Upload the trained local sub-models and aggregate them to update the master model.
Finally, download the master model together with the choice keys of all parents and offspring to all clients to evaluate the test errors and FLOPs. Upload the test errors and FLOPs to the server and calculate the weighted averaging of test errors for each individual.
Once the objective values of all parent and offspring individuals are calculated, environmental selection can be carried out to generate the parent individuals of the next generation based on the elitist non-dominated sorting and crowding distance, as discussed in Algorithm 2 in Section II.C.
In the following generations, similar steps as described above will be carried out, except that the master model is updated only once at each generation after the global model (also a sub-model randomly sampled from the master model) of all offspring individuals is trained on the sampled clients and the resulting local sub-models are aggregated. It should be emphasized that at each generation, the master model shared by all individuals need to be downloaded to all participating clients only once for fitness evaluations.
Note that model aggregation is different from that in the conventional federated learning. The reason is that different clients may be sampled for different individuals, and different individuals may have different model structures, which cannot be directly aggregated. Fig. 6 shows an illustrative example, where the master model has two choice blocks and . There are two individuals, and the choice keys for the two individuals are [0, 0, 0, 1] and [1, 0, 1, 1], respectively. The resulting sub-models are and , and and . We further assume that client 1 is chosen for training sub-model and and client 2 for and . Then each client updates its model parameters according to the available local dataset and then upload the trained model to the server, which is denoted by the shaded square. And then two master models are reconstructed by filling in the sub-models that are not updated, which are denoted by the white squares. Since the reconstructed master models have the same structure, they can be easily aggregated using i.e., the weighted averaging. The pseudo code for model aggregation is presented in Algorithm 3.
The advantage of the above filling and aggregation method is able to prevent abrupt changes in some sub-models and improve convergence in federated learning. In addition, this aggregation method does not require extra communication resources, since this operation is only performed on the server.
From the above description, we can see that the double-sampling strategy fits perfectly well with the population based evolutionary search so that the objective values of all individuals at one generation can be evaluated within one communication round, seamlessly embedding a generation of architecture search into one round of training in federated learning.
Iii-C Overall Framework
The fitness evaluations are done for both parent and offspring populations at every generation, which is equivalent to a communication round in the proposed real-time evolutionary NAS. For fitness evaluations, both the master model and all choice keys are downloaded to all participating clients. Thus, we do not need to download any model parameters for training of any sub-models in the next round and it is sufficient to download the choice keys only (refer to Lines 32-33 in Algorithm 4. After that, each client uploads the updated local sub-models to the server for model aggregation. As a result, the proposed model sampling method can reduce both local computation complexity and communication costs for uploading the models.
It should be noticed that the parent sub-models are trained only at the first generation. In the subsequent evolutionary optimization, only offspring sub-models need to be trained. However, all sub-models sampled by need to be evaluated to calculate the test errors for fitness evaluations at each generation, because training the offspring sub-models will also affect the parameters of the parent sub-model since parent and offspring sub-models always share weights of the master model. In addition, we do not need to reinitialize the model parameters of the sampled offspring sub-models before training starts. Due to the client sampling strategy, the population size does not affect the communication costs for fitness evaluations, since the entire master model is downloaded from the server and sampling of the master model can be done on the clients.
Since NSGA-II is a Pareto based multi-objective optimization algorith which can find a set of optimal models that trade off between accuracy and computational complexity (FLOPs). Therefore, for online applications, the user needs to articulate preferences to select one of the Pareto optial solutions from the parent population in each round. In practice, the Pareto solutions with the high test accuracies or those near the knee points [51, 50, 53] are preferred unless there are other strong user-specified preferences.
Iv Experimental Results
Iv-a NSGA-II Settings
The settings for NSGA-II are listed in the Table I. Here, we use binary one-point crossover and binary bit flip mutation as the genetic operators. Since the master network only contains four choice blocks, a two-bit binary string is used for representation.
Iv-B Federated Learning Settings
The hyper parameters for federated learning are presented in Table II.
|Total Clients||10, 20, 50|
|Train Batch Size||50|
|Test Batch Size||100|
|Initial Learning rate||0.1|
Here, learning decay means the decay of the learning rate over each communication round. Apart from this, the number of total communication rounds is not set here, because it is equal to the number of generations in real-time evolutionary optimization.
Iv-C Model and Dataset Settings
The master model used in this work is a deep neural network with multiple branches containing a total of 28 layers (12 choice blocks, each containing two convolution layers). The number of channels for all block layers are [64, 64, 64, 128, 128, 128, 256, 256, 256, 512, 512, 512]. The overall structure of the master model is shown in Fig. 3. Apart from this, the trainable parameters in the batch normalization layer may slow down the convergence speed in federated learning, since they perform poorly for learning with small batch sizes  and weight sharing training paradigms [52, 14, 10], especially for non-IID scenarios. In addition, we disable both the variables and exponential moving average variables in the batch normalization layer because we found that they may cause the divergence of the master model.
We adopt Cifar10  as the dataset, which contains 50000 training and 10000 testing 32x32 RGB images with 10 different kinds of objects. For IID federated simulations, all training image data are evenly and randomly distributed to each local client without overlaps. For experiments on non-IID data, each client has images with different kinds of objects. We do not consider very extreme cases where each client has data with only or classes, since this is not realistic in the real world environment. For instance, it is not beneficial to collaboratively train a global model with clients having completely different datasets.
Note that we do not apply any data augmentation  in our simulation, since the server cannot do any operations on the client data in federated learning.
In all experiments, we use one GTX 1080Ti GPU.
Iv-D Experiment Results
Iv-D1 Baseline Model Used in Federated Learning
We use ResNet18 as the baseline model and its parameters are provided in Table III. As previously mentioned, all trainable parameters in the batch normalization layers are removed, resulting in a total FLOPs (MAC) of 0.5587G on the cifar10 dataset. The settings for the federated learning follow those given in Section III.
|Layer Name||Output Size||18 Layers|
Iv-E Federated Evolutionary NAS Results
We adopt three different numbers of clients (10, 20 and 50) for real-time multi-objective evolutionary federated NAS and the obtained Pareto optimal solutions on both IID and non-IID data after 500 rounds (generations) of optimization are shown in Fig. 8. From these results we can see that the real-time evolutionary federated NAS algorithm is able to achieve a set of evenly distributed Pareto optimal solutions.
The following two observations can be made. First, the classification accuracies of the optimized models on the IID data are better than those for on the non-IID data. Second, the smaller the number of the clients, the better the classification performance. These two phenomenons are reasonable, since learning on IID data is much easier than learning on non-IID data in federated learning. On the other hand, the more the number of clients, the less data there is on each client, as the amount of data in total is given.
To take a closer look at the obtained models, we present the test accuracies and FLOPs of the model having the highest test accuracy (called High) and the knee solution (called Knee), as indicated in Fig. 8, as well as that of the ResNet in Table IV.
|Model||clients||IID||Test Accuracy||FLOPs (MAC)|
From these results, we can see that federated learning with a total of and clients, the model having the highest accuracy found by the proposed algorithm is better than the original ResNet in accuracy and has a much lower computational complexity. For 10 clients, the knee solutions found by the proposed algorithm have a much lower model FLOPS than the original ResNet. By contrast for 20 clients, the model FLOPs of the knee solutions is again must lower than that of the ResNet and the performance on both IID and non-IID data is better. However, for clients, the test accuracies of the models found by the proposed algorithm is worse than the ResNet, although the model FLOPs are lower. This indicates that it becomes harder to find an optimal global model as the amount of data on each client becomes less.
Iv-F Real-time Performance
Since the proposed method is meant for real-time purposes, here we examine the performance of two models during the optimization, one is the model having the highest test accuracy and the other is the knee solution. For simplicity, we only investigate the real-time performance when the number of participating clients is , which is presented in Fig. 9. For comparison, the performance of ResNet18 is also plotted. We can see clearly that ResNet18 performs better than both models found by the evolutionary search method at the early stage. However, the two solutions are able to outperform ResNet after approximately communication rounds. We can also find that the performance of the best model is very stable during the evolutionary search, although the knee solution experiences some minor fluctuations in performance. Both models perform much more stably than those in the conventional offline evolutionary NAS in .
The model FLOPs of the two solutions are shown in Fig. 9(c)(d), which are smaller than that of the original ResNet18.
From the above results, we can see that the proposed real-time evolutionary NAS algorithm is not only able to find light weighted models, but also ensure stable and competitive performance during the optimization.
Iv-G Comparison with Offline Federated NAS
Here, we also compare the proposed algorithm with the offline evolutionary federated NAS optimization. In the conventional offline model, the sampled model is downloaded to all clients with the same structure and offspring models should be reinitialized and trained from scratch, similar to the settings in .
For a fair comparison, models are collaboratively trained with one communication round for fitness evaluations. The number of clients is set to be and the Pareto optimal solutions obtained on the non-IID data after 50 generations are plotted in Fig. 10. Only four solutions are found in the last generation. Finally, the model having the highest accuracy and the knee solution as indicated in Fig. 10 are selected to be trained from scratch for communication rounds and their learning curve is shown in Fig. 11. From these results we can see that the performance of the best model found by the offline evolutionary NAS is lower than 75 in this case, which is attributed to the very fast convergence in the early stage of the evolutionary search. These results also imply that in the offline evolutionary NAS, it is non-trivial to define the number of training rounds at each generation.
It takes about hrs/gpu for the real-time evolutionary NAS to run 500 generations (rounds). In contrast, the offline evolutionary NAS for 50 generations needs about hrs/gpu without considering the re-training time. This means that on average, the real-time evolutionary NAS is approximately five times faster than the offline evolutionary NAS algorithm for each round of search.
V Conclusions and Future Work
This paper proposes a real-time multi-objective evolutionary method for federated NAS, which can effectively avoid extra communication costs and computational resources and maintain a stable performance of the models during the optimization. This is achieved by a double-sampling approach that samples a master model shared by all individuals in the same population and samples the participating clients for training the global model. This way, one generation of evolutionary optimization can be embedded and completed within one single communication round, thereby reducing both communication and computation costs.
The experimental results demonstrate that the proposed evolutionary federated NAS framework is able to find a set of evenly distributed Pareto optimal solutions for both IID and non-IID datasets. Among these Pareto optimal solutions, we can obtain models having different architectures that present a trade-off between classification performance and computational complexity. In addition, we show that these models are computationally much simpler than the standard model while the performance is still highly competitive. The high computational efficiency of the proposed real-time evolutionary NAS is also confirmed by the comparative results with the conventional offline evolutionary method.
The present work is a first valuable step towards the application of neural architecture search to the federated learning framework. In the future, we are going to verify and extend the the proposed algorithm for real-time NAS in large-scale federated learning systems. In addition, new techniques remain to be developed to deal with data that are vertically partitioned and distributed on the clients.
-  (2019) Federated collaborative filtering for privacy-preserving personalized recommendation system. Cited by: §I.
-  (1994) An evolutionary algorithm that constructs recurrent neural networks. 5 (1), pp. 54–65. Cited by: §II-B.
-  (2016) Designing neural network architectures using reinforcement learning. Cited by: §II-B.
-  (2018) Understanding and simplifying one-shot architecture search. Cited by: §II-B.
-  (1991) Stochastic gradient learning in neural networks. 91 (8), pp. 12. Cited by: §II-A2.
-  (2010) Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT’2010, pp. 177–186. Cited by: §II-B.
-  (2018) Expanding the reach of federated learning by reducing client resource requirements. Cited by: §I.
-  (2019) Communication-efficient federated deep learning with asynchronous model update and temporally weighted aggregation. Cited by: §I.
-  (2017) Xception: deep learning with depthwise separable convolutions. In , pp. 1251–1258. Cited by: §III-A, §III-A.
-  (2019) Fairnas: rethinking evaluation fairness of weight sharing neural architecture search. Cited by: §IV-C.
-  (2005) Multi-objective optimization using evolutionary algorithms. Wiley. Cited by: §II-C.
-  (2000) A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: nsga-ii. In International conference on parallel problem solving from nature, pp. 849–858. Cited by: §II-C, §II-C.
-  (2019) Searching for a robust neural architecture in four gpu hours. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1761–1770. Cited by: §I, §I, §II-B.
-  (2019) Single path one-shot neural architecture search with uniform sampling. Cited by: §II-B, §IV-C.
-  (2017) Deep pyramidal residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5927–5935. Cited by: §III-A.
-  (2015) Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. Cited by: §I.
-  (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778. Cited by: §I, §II-B, §III-A.
-  (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. Cited by: §III-A, §III-A.
-  (2017) Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708. Cited by: §II-B.
-  (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. Cited by: §I, §III-A.
-  (2017) Batch renormalization: towards reducing minibatch dependence in batch-normalized models. In Advances in neural information processing systems, pp. 1945–1953. Cited by: §IV-C.
-  Y. Jin (Ed.) (2006) Multi-objective machine learning. Springer. Cited by: §II-C.
-  (2018) Neural architecture search with bayesian optimisation and optimal transport. In Advances in Neural Information Processing Systems, pp. 2016–2025. Cited by: §II-B.
-  (2016) Federated learning: strategies for improving communication efficiency. Cited by: §I.
-  () CIFAR-10 (canadian institute for advanced research). IEEE Transactions on Evolutionary ComputationarXiv preprint arXiv:1712.04621arXiv preprint arXiv:1602.05629ACM Transactions on Intelligent Systems and Technology (TIST)arXiv preprint arXiv:1610.05492arXiv preprint arXiv:1510.00149arXiv preprint arXiv:1812.07210arXiv preprint arXiv:1903.07424IEEE transactions on neural networks and learning systemsnaturearXiv preprint arXiv:1808.05377arXiv preprint arXiv:1802.03268The handbook of brain theory and neural networksarXiv preprint arXiv:1502.03167arXiv preprint arXiv:1611.01578arXiv preprint arXiv:1611.02167Naval Research Logistics (NRL)IEEE Transactions on Signal ProcessingIEEE transactions on Neural NetworksEvolutionary computationProceedings of the IEEEarXiv preprint arXiv:1711.00436arXiv preprint arXiv:1806.09055arXiv preprint arXiv:1910.11858Proceedings of the IEEEProceedings of Neuro-NımesIEEE Transactions on Evolutionary ComputationarXiv preprint arXiv:1704.04861IEEE Transactions on knowledge and data engineeringarXiv preprint arXiv:1904.00420arXiv preprint arXiv:1907.01845arXiv preprint arXiv:1812.08928IEEE Transactions on CyberneticsIEEE Transactions on Evolutionary ComputationIEEE Transactions on CyberneticsIEEE Transactions on Evolutionary ComputationarXiv preprint arXiv:1901.09888. External Links: Cited by: §IV-C.
-  (2012) Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105. Cited by: §II-B.
-  (2015) Deep learning. 521 (7553), pp. 436–444. Cited by: §II-B.
-  (1995) Convolutional networks for images, speech, and time series. 3361 (10), pp. 1995. Cited by: §II-B.
-  (2017) Hierarchical representations for efficient architecture search. Cited by: §I, §I, §II-B.
-  (2018) Darts: differentiable architecture search. Cited by: §I, §I, §II-B.
-  (2016) Communication-efficient learning of deep networks from decentralized data. Cited by: §I.
-  (2009) A survey on transfer learning. 22 (10), pp. 1345–1359. Cited by: §I.
-  (2017) The effectiveness of data augmentation in image classification using deep learning. Cited by: §IV-C.
-  (2018) Efficient neural architecture search via parameter sharing. Cited by: §I, §I, §II-B, §II-B, §II-B.
Regularized evolution for image classifier architecture search. In
Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, pp. 4780–4789. Cited by: §I, §I, §II-B.
-  (2017) Large-scale evolution of image classifiers. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 2902–2911. Cited by: §I, §II-B.
-  (2018) Mobilenetv2: inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520. Cited by: §III-A.
-  (1997) Bidirectional recurrent neural networks. 45 (11), pp. 2673–2681. Cited by: §II-B.
-  (2015) Taking the human out of the loop: a review of bayesian optimization. 104 (1), pp. 148–175. Cited by: §II-B.
-  (2015) Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp. 1310–1321. Cited by: §I.
A genetic programming approach to designing convolutional neural network architectures. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 497–504. Cited by: §I, §II-B.
Surrogate-assisted evolutionary deep learning using an end-to-end random forest-based performance predictor. Note: DOI: 10.1109/TEVC.2019.2924461 Cited by: §I.
Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-First AAAI Conference on Artificial Intelligence, Cited by: §II-B.
-  (2015) Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9. Cited by: §II-B.
-  (2016) Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818–2826. Cited by: §II-B.
-  (2019) Mnasnet: platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820–2828. Cited by: §I.
-  (2010) Transfer learning. In Handbook of research on machine learning applications and trends: algorithms, methods, and techniques, pp. 242–264. Cited by: §I.
A linear relaxation heuristic for the generalized assignment problem. 39 (2), pp. 137–151. Cited by: §II-B.
-  (1999) Evolving artificial neural networks. 87 (9), pp. 1423–1447. Cited by: §II-B.
-  (2018-07) A method for a posteriori identification of knee points based on solution density. In 2018 IEEE Congress on Evolutionary Computation (CEC), Vol. , pp. 1–8. External Links: Cited by: §III-C.
-  (2019) Benchmark problems and performance indicators for search of knee points in multiobjective optimization. (), pp. 1–14. External Links: Cited by: §III-C.
-  (2018) Slimmable neural networks. Cited by: §IV-C.
-  (2015-12) A knee point-driven evolutionary algorithm for many-objective optimization. 19 (6), pp. 761–776. External Links: Cited by: §III-C.
-  (2018) Practical block-wise neural network architecture generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2423–2432. Cited by: §II-B.
-  (2019) Multi-objective evolutionary federated learning. Cited by: §I, §I, §III-B, §IV-F, §IV-G.
-  (2016) Neural architecture search with reinforcement learning. Cited by: §I, §II-B.
-  (2018) Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697–8710. Cited by: §II-B.