Besides end-to-end learning a black-box neural network, in this paper, we propose a new deep-learning methodology, i.e. network transplanting. Instead of learning from scratch, network transplanting aims to merge several convolutional networks that are pre-trained for different categories and tasks to build a generic, distributed neural network.
Network transplanting is of special values in both theory and practice. We briefly introduce key deep-learning problems that network transplanting deals with as follows.
1.1 Future potential of learning a universal net
Instead of learning different networks for different applications, building a universal net with a compact structure for various categories and tasks is one of ultimate objectives of AI. In spite of the gap between current algorithms and the target of learning a huge universal net, it is still meaningful for scientific explorations along this direction. Here, we list key issues of learning a universal net, which are not commonly discussed in the current literature of deep learning.
The start-up cost w.r.t. sample collection is also important, besides the total number of training annotations. Traditional methods usually require people to simultaneously prepare training samples for all pairs of categories and tasks before the learning begins. However, it is usually unaffordable, when there is a large number of categories and tasks. In comparison, our method enables a neural network to sequentially absorb network modules of different categories one-by-one, so the algorithm can start without all data.
Massive distributed learning & weak centralized learning: Distributing the massive computation of learning the network into local computation centers all over the world is of great practical values. There exist numerous networks locally pre-trained for specific tasks and categories in the world. Centralized network transplanting physically merges these networks into a compact universal net with a few or even without any training samples.
Delivering models or data: Our delivering pre-trained networks to the computation center is usually much cheaper than collecting and sending raw training data in practice.
Middle-to-end semantic manipulation for application: How to efficiently organize and use the knowledge in the net is also a crucial problem. We use different modules in the network to encode knowledge of different categories and that of different tasks. Like building LEGO blocks, people can manually connect a category module and a task module to accomplish a certain application (see Fig. 1(left)).
|Annotation cost||Sample preparation||Interpretability||Catastrophic forgetting||Modular manipulate||Optimization|
|Directly learning a multi-task net||Massive||Simultaneously prepare samples for all tasks and categories||Low||–||Not support||back prop.|
|Transfer- / meta- / continual-learning||
Some support weakly-supervised learning
|Some learn a category/task after another||Usually low||Most algorithmically alleviate||No support||back prop.|
|Transplanting||A few or w/o annotations||Learns a category after another||High||Physically avoid||Support||back-back prop.|
1.2 Task of network transplanting
To solve above issues, we propose network transplanting, i.e. building a generic model by gradually absorbing networks locally pre-trained for specific categories and tasks. We design an interpretable modular structure for a target network, namely a transplant net, where each module is functionally meaningful. As shown in Fig. 1(left), the transplant net consists of three types of modules, i.e. category modules, task modules, and adapters. Each category module extracts general features for a specific category (e.g. the dog). Each task module is learned for a certain task (e.g. classification or segmentation) and is shared by different categories. Each adapter projects output features of a category module to the input space of a task module. Each category/task module is shared by multiple tasks/categories.
We can learn an initial transplant net with very few tasks and categories in the scenario of traditional multi-task/category learning. Then, we gradually grow the transplant net to deal with more categories and tasks via network transplanting. Network transplanting can be conducted with or without human annotations as additional supervision. We summarize two typical types of transplanting operations in Fig. 1(right). The core technique is to learn an adapter to connect a task module in the transplant net and a category module from another network111Please see supplementary materials for details..
The elementary transplanting operation is shown in Fig. 2(left). We are given a transplant net with a task module that is learned to accomplish a certain task for many categories, except for the category . We hope the task module to deal with the new category , so we need another network (namely, a teacher net) with a category module and a task module . The teacher net is pre-trained for the same task on the category . We may (or may not) have a few training annotations of category for the task. Our goal is to transplant the category module in the teacher net to the transplant net.
Note that we just learn a small adapter module to connect to . We do not fine-tune and during the transplanting process to avoid damaging their generality.
However, learning adapters but fixing parameters of category and task modules proposes specific challenges to deep-learning algorithms. Therefore, in this study, we proposed a new algorithm, namely back distillation, to overcome these challenges. The back-distillation algorithm uses the cascaded modules of the adapter and to mimic upper layers of the pre-trained teacher net. This algorithm requires the transplant net to have similar gradients/Jacobian with the teacher net w.r.t. ’s output features for distillation. In experiments, our back-distillation method without any training samples even outperformed the baseline with 100 training samples (see Table 2(left)).
1.2.1 Difference to previous knowledge transferring
Although most transfer-learning algorithms cannot be directly used to solve core problems mentioned in Section1.1, the proposed network transplanting is close to the spirit of continual learning (or lifelong learning) [16, 6, 19, 27, 23]. As an exploratory research, we summarize our essential differences from traditional studies in Table 1.
Modular interpretability more controllability: Besides the discrimination power, the interpretability is another important property of a neural network, which has received increasing attention in recent years. In particular, traditional transfer learning and knowledge distillation that are implemented in a black-box manner [16, 5], so the generalization process requires careful control. Whereas, our transplant net clarifies the functional meaning of each intermediate network module, which makes the knowledge-transferring process more controllable. I.e. the interpretable structure clearly points out network modules that are related to the target application.
Bottleneck of transferring upper modules11footnotemark: 1: Most deep-learning strategies are not suitable to directly transfer pre-trained upper modules, including (i) directly optimizing the task loss on the top of the network and (ii) traditional distillation methods . These methods mainly transfer low-layer features and learn new upper modules to reuse these features, rather than directly transfer pre-trained upper modules. In contrast, network transplanting just allows us to modify the lower adapter, when we transfer a pre-trained upper task module to a new category. It is not permitted to modify the upper module. This requirement physically avoids the catastrophic forgetting, but it is difficult to optimize a lower adapter if the upper is fixed (see Section 3.1 for theoretical analysis). Meanwhile, it is difficult to distill knowledge from the teacher net to the adapter. It is because except for the final network output, high-layer features in and those in the teacher net are not semantically aligned.
Thus, in this paper, the proposed back distillation first breaks the bottleneck of transferring upper modules.
Catastrophic forgetting: Continually learning new jobs without hurting the performance of old jobs is a key issue for continual learning [16, 19]. Our method exclusively learns the adapter to physically prevent the learning of new categories from changing existing modules. Furthermore, when a transplant net has been constructed, we can optionally fine-tune a task/category module based on different categories/tasks to ensure the network generality.
We can summarize contributions of this study as follows. (i) We propose a new deep-learning method, network transplanting with a few or even without additional training annotations, which can be considered as a theoretical solution to three issues in Section 1.1. (ii) We develop an optimization algorithm, i.e. back-distillation, to overcome specific challenges of network transplanting. (iii) Preliminary experiments proved the effectiveness of our method. Our method significantly outperformed baselines.
2 Related work
Because network transplanting is a new concept in machine learning, we would like to discuss its connections to different state-of-the-art algorithms. Firstly, we propose a new modular structure for networks, which disentangles a black-box network into different meaningful modules. Similarly, some studies have explored new representation structures instead of neural networks, such as forests and decision trees[12, 32, 7, 25] and automatic learning of optimal network structures [33, 14, 34, 31].  learned a large modular neural network with thousands of sub-networks.
Interpretability: Unlike above studies, our dividing a network into functionally meaningful modules makes the structure interpretable. Other studies of enhancing network interpretability mainly either learn disentangled features filters/capsules in middle layers [30, 26, 17] or learn meaningful input codes of generative nets [3, 10].
Meta-learning & transfer learning: Meta-learning [4, 1, 13] aims to extract generic knowledge shared by different tasks/categories/models to guide the learning of a specific model. Transfer-learning methods transfer network knowledge through categories  or datasets . Especially, continual learning [16, 6, 19, 27, 23] transfers knowledge from previous tasks to guide the new task. [16, 27] expanded the network structure during the learning process. In contrast, our study defines modular network structures with strict semantics, which allows people to semantically control the knowledge to transfer. Meanwhile, network transplanting physically avoids the catastrophic forgetting. In addition, our back distillation method solves the challenge of transferring upper modules, which is different from traditional transferring of low-layer features.
Distillation:  proposed the knowledge distillation to transfer knowledge between networks. Some recent studies [7, 25] distilled network knowledge into decision trees.  proposed an online distillation method to efficiently learn distributed networks. [29, 22] distilled the attention distribution/Jacobian from the teacher network to the student network, which is related to our back-distillation technique. However, these Jacobian distillation methods are hampered considering specific challenges of network transplanting (see Section 3.2 and the appendix for details). To overcome these challenges, in this study, we design a new method of back-distillation, which uses pseudo-gradients instead of using real gradients for distillation. We also balance magnitudes of neural activations between two networks, which is necessary for network transplanting.
3 Algorithm of network transplanting
Overview: As shown in Fig. 2(left), we are given a teacher net for a single or multiple tasks w.r.t. a certain category . Let the category module in the bottom of the teacher net have layers, and it connects a specific task module in upper layers. We are also given a transplant net with a generic task module , which has been learned for multiple categories except for the category .
The initial transplant net with a task module (before transplanting) can be learned via traditional scenario of learning from samples of some categories. We can roughly regard to encode generic representations for the task. Similarly, the category module extracts generic features for multiple tasks. Thus, we do not fine-tune or to avoid decreasing their generality.
Our goal is to transplant to by learning an adapter with parameters , so that the task module can deal with the new category module .
The basic idea of network transplanting is that we use the cascaded modules of and to mimic the specific task module in the teacher net. We call the transplant net a student net. Let denote the output feature of the category module given an image , i.e. . and are given as outputs of the teacher net and the student net, respectively. Thus, network transplanting can be described as
3.1 Problem of space projection & back-distillation
It is a challenge to let an adapter project the output feature space of properly to the input feature space of . The information bottleneck theory [24, 18] shows that a network selectively forgets certain space of middle-layer features and gradually focuses on discriminative features during the learning process (see Fig. 3(left)). Thus, both the output of and the input of have vast forgotten space. Features in the forgotten input space of
cannot pass most feature information through ReLU layers inand reach . The forgotten output space of is referred to the space that does not contains ’s output features.
Vast forgotten feature spaces significantly boost difficulties of learning. Since valid input features of usually lie in low-dimensional manifolds, most features of the adapter fall inside the forgotten space. I.e. will not pass most information of input features to network outputs. Consequently, the adapter will not receive informative gradients of the loss for learning.
Fig. 3(right) illustrates ideal and problematic projections. A typical problematic projection is to project a feature to a forgotten input space of . Another typical problem is many-to-one projections, which limit the diversity of features and decrease the representation capability of the student net. More crucially, initial many-to-one projections significantly affect the further learning process, because the back-propagation takes current space projections as anchors to fine-tune the network.
To learn good projections, we propose to force the gradient (also known as attention, Jacobian) of the student net to approximate that of the teacher, which is a necessary condition of .
where is an arbitrary function of that outputs a scalar. denote parameters of the adapter . Therefore, we use the following distillation loss for back-distillation:
where is the task loss of the student net; denotes the ground-truth label; is a scaling scalar. This formulation is similar to the Jacobian distillation in . We omit , if we learn the adapter without additional training labels.
3.2 Learning via back distillation
It is difficult for most recent techniques, including those for Jacobian distillation11footnotemark: 1, to directly optimize the above back-distillation loss. We briefly analyze the difficulties as follows. The minimization of the distillation loss is actually to push gradients of the student net towards those of the teacher net. To simplify the notation, we use and to denote gradients w.r.t. the feature map in the student net and the teacher net, respectively. As shown in Eqn. (4a), the computation of (or ) is sensitive to feature maps of layers in the and (those in ). Thus, it requires the student network to yield well-optimized feature maps to enable an effective distillation process. However, it is a chicken-and-egg problem—distilling optimal parameters and generating optimal feature maps of middle layers: Chaotic initial feature maps hurt the capability of distilling knowledge into , but the feature maps are produced using .
To overcome the optimization problem, we need to make gradients of agnostic with regard to feature maps. Thus, we propose two pseudo-gradients to replace in the loss, respectively. The pseudo-gradients follow the paradigm in Eqn. (4b).
where we define . Just like in Eqn. (2), we assume . , each is the derivative of the layer function for back-propagation. denotes a set of feature maps of all middle layers, and is the feature map of the -th layer.
In Eqn. (4b), we make the following revisions to the computation of gradients, in order to make gradients agnostic with regard to
. We ignore dropout operations and replace derivatives of max-poolingwith derivatives of average-pooling . We also revise the derivative of the ReLU to either or , where is a random feature map; denotes the element-wise product. For each input image, we set the same random feature map and initial gradients for both and to make and comparable with each other. Above revisions are made for to ease the back distillation, and they are not related to the computation of the task loss .
In this way, we conduct the back-distillation algorithm by . The distillation loss can be optimized by propagating gradients of gradient maps to the upper layers, and we consider this as back-back-propagation11footnotemark: 1. Tables 2 and 3 have exhibited the superior performance of the back-back-propagation.
Computation of : According to Eqn. (2), can be any arbitrary function. Thus, we can enumerate functions of by randomizing different values of . We use each to produce a pair of and
for back distillation. For the task of object segmentation, the output is a tensor, where and denote the height and width of the output image, and indicates the number of segmentation labels. For each image, we randomly sample . For the task of single-category classification, the output
is a scalar. Nevertheless, we can still generate a random matrix() for each image ( in experiments), which produces two enlarged pseudo-gradient maps for back distillation11footnotemark: 1. We normalize to the ranges of , which ensures a stable distillation in experiments.
To simplify the story, we limit our attention to testing network transplanting operations. We do not discuss other related operations, e.g. the fine-tuning of category and task modules and the case in Fig. 1(a), which can be solved via traditional learning strategies.
We designed three experiments to evaluate the proposed method. In Experiment 1, we learned toy transplant nets by inserting adapters between middle layers of pre-trained CNNs. Then, Experiments 2 and 3 were designed considering the real application of learning a transplant net with two task modules (i.e. modules for object classification and segmentation) and multiple category modules. As shown in Fig. 4(b,d), we can divide the entire network-transplanting procedure into an operation sequence of transplanting category modules to the classification module and another operation sequence of transplanting category modules to the segmentation module. Therefore, we separately conducted the two sequences of transplanting operations in Experiments 2 and 3 for more convincing results.
Because our back-distillation strategy decreases the demand for training samples, we tested the learning of adapters with limited numbers of samples (i.e. 10, 20, 50, and 100 samples). We even tested network transplanting without any training samples in Experiment 1, i.e. optimizing the distillation loss without considering the task loss.
Baselines: We compared our back-distillation method (namely back-distill) with two baselines. All baselines exclusively learned the adapter without fine-tuning the task module for fair comparisons. The first baseline only optimized the task loss without distillation, namely direct-learn. The second baseline is the traditional distillation , namely distill, where the distillation loss is . The distillation was applied to outputs of task modules and , because except for outputs, other layers in and did not produce features on similar manifolds. We tested the distill method in object segmentation, because unlike single-category classification, segmentation outputs had correlations between soft output labels.
Network structures, datasets, & details: We transplanted category modules to a classification module in the first two experiments, and transplanted category modules to a segmentation module in the third experiment. In recent studies, people usually extended the structure of widely-used VGG-16 net  to implement classification [21, 30] and segmentation , as standard baselines of the two tasks. Thus, as shown in Fig. 4(a), we can represent a teacher net for both classification and segmentation as a network with a single category module and two task modules. The network branch for classification was exactly a VGG-16 net, and the network branch for segmentation was identical to the FCN-8s model proposed in . Because the first five layers of the FCN-8s and those of the VGG-16 share the same structure, we considered the first five layers (including two conv-layers, two ReLU layers, and one pooling layer) as the shared category module and regarded upper layers of the VGG-16 and the FCN-8s as two task modules. Both branches are benchmark networks.
We followed standard experimental settings in  to learn the FCN-8s for each category, which used the Pascal VOC 2011 dataset (with segmentation labels on 8498 PASCAL images collected by ). For object classification, we followed standard settings in  that used the PASCAL VOC images to learn CNNs for the binary classification of a single category from random images. Note that we only learned and merged teacher networks for five mammal categories, i.e. the cat, cow, dog, horse, and sheep categories. Mammal categories share similar object structures, which make features of a category transferable to other categories.
Now, we introduce adapter structures, as shown in Fig. 4(c). An adapter contained conv-layers, each followed by a ReLU layer ( or in experiments)222Each conv-layer in the adapter contained filters. Each filter was a tensor with a in Experiment 1 or a tensor without padding in Exps. 2 and 3, to avoid changing the size of feature maps, where
tensor without padding in Exps. 2 and 3, to avoid changing the size of feature maps, whereis the channel number of .. In addition, we inserted a “reorder” layer and a “re-scaling” layer11footnotemark: 1 in front of conv-layers in the adapter. The “reorder” layer randomly reordered channels of the features from the category module, which enlarged the dissimilarity between output features of different category modules. We inserted the “reorder” layer to mimic feature states in real applications for fair evaluation. The “re-scaling” layer normalized the scale of features from the category module, i.e. , for robust network transplanting. is a fixed scalar. and denote the image set of the new category and the image set of categories that had been already modeled by the transplant net, respectively333Because we used the task module in the dog network as the generic task module , we got .. denotes the input feature of given an image . denotes the Frobenius norm. We set the parameter .
Insert one conv-layer
|# of samples||cat||cow||dog||horse||sheep||Avg.||
Insert three conv-layers
|# of samples||cat||cow||dog||horse||sheep||Avg.|
4.1 Exp. 1: Adding adapters to pre-trained CNNs
In this experiment, we conducted a toy test, i.e. inserting and learning an adapter between a category module and a task module to test network transplanting. Here, we only considered networks with VGG-16 structures  for single-category classification. These networks were strongly supervised using all training samples and achieved error rates of 1.6%, 0.6%, 4.1%, 1.6%, and 1.0% for the classification of the cat, cow, dog, horse, and sheep categories, respectively. Then, we learned two types of adapters (see Fig. 4(c)), which have been introduced before.
Because the classification output is a scalar without neighboring outputs to provide correlations, we simply set without any gradient randomization. Instead, we used the revised dummy ReLU operations in Eqn. (4b) to ensure the value diversity of for learning. More specifically, we used to compute expedient derivatives of ReLU operations in the task module, and used in the adapter444All derivative functions in Eqn. (4b) are only used for distillation, which will not affect gradient propagations from .. We set for object classification in Experiments 1 and 2.
In Fig. 5, we compared the space of fc8 features, when we used our method and the direct-learn baseline, respectively, to learn the adapter with three conv-layers. The adapter learned by our method passed much stronger information to the final fc8 layer and yielded more diverse features. It demonstrates that our method better avoided problematic projections in Fig. 3 than the direct-learn baseline.
|Experiment 2: transplanting to a classification module|
|# of samples||cat||cow||horse||sheep||Avg.||# of samples||cat||cow||horse||sheep||Avg.|
|Experiment 3: transplanting to a segmentation module|
|# of samples||cat||cow||horse||sheep||Avg.||# of samples||cat||cow||horse||sheep||Avg.|
In Table 2, we compared our back-distill method with the direct-learn baseline when we inserted an adapter with a single conv-layer and when we inserted an adapter with three conv-layers. Table 2(left) shows that compared to the – error rates of the direct-learn baseline, our back-distill method yielded a significant lower classification error (–). Even without any training samples, our method still outperformed the direct-learn method with 100 training samples.
Note that given an adapter with multiple conv-layers (e.g. three conv-layers) without any training samples, our back-distill method was not powerful enough to learn the adapter. Because deeper adapters with more parameters had more flexibility in representation, which required stronger supervision to avoid over-fitting. For example, in the last row of Table 2, our method successfully optimized an adapter with a single conv-layer (the error rate was 1.75%), but was hampered when the adapter had three conv-layers (the error rate was 49.9%, and our method produced a biased short-cut solution).
4.2 Exp. 2: Operation sequences of transplanting category modules to the classification module
In this experiment, we evaluated the performance of transplanting category modules to the classification module. We considered the classification module of the dog555Because the dog category contained more training samples, the CNN for the dog was believed to be better learned. Thus, we used the task module learned for dog images as a generic task module. as a generic one. We transplanted category modules of other four mammal categories to this task module. According to the experience in Experiment 1, we set the adapter to contain three conv-layers. Following Eqn. (4b), we used the operation to compute derivatives of ReLU operations in both the task module and the adapter33footnotemark: 3. The only exception was the lowest ReLU operation of the task module, for which we applied . We generated for each input image by concatenating matrices along the third dimension, where contained 20%/80% positive/negative elements. The generation of is introduced in Section 3.2.
Table 3 shows the performance when we transplanted the category module to a task module oriented to categories with similar structures. We tested our method with a few (10–100) training samples. When there were more than 50 training samples, our method yielded about a half classification error of the direct-learn baseline.
|# of samples||cat||cow||horse||sheep||Avg.||# of samples||cat||cow||horse||sheep||Avg.|
4.3 Exp. 3: Operation sequences of transplanting category modules to the segmentation module
In this experiment, we evaluated the performance of transplanting category modules to the segmentation module. Five FCNs were strongly supervised using all training samples for single-category segmentation. These networks achieved pixel-level segmentation accuracies (defined in ) of 95.0%, 94.7%, 95.8%, 94.6%, and 95.6% for the cat, cow, dog, horse, and sheep categories, respectively. Like in Experiment 2, we considered the segmentation module of the dog55footnotemark: 5 as a generic one. We transplanted category modules of other four mammal categories to this task module. According to the experience in Experiment 1, we set the adapter to contain one conv-layer. Following Eqn. (4b), we used the operation to compute derivatives of ReLU operations33footnotemark: 3. We set for all categories in this experiment.
Table 3 compares pixel-level segmentation accuracy between our method and the direct-learn baseline. We tested our method with a few (10–100) training samples. Our method exhibited 10%–12% higher accuracy than the direct-learn baseline.
4.4 Transplant to similar or dissimilar categories?
Theoretically, just like transfer learning, the task module should deal with a set of categories that have similar structures with the new category to ensure a high efficiency. In fact, identifying categories with similar structures is still an open problem. People manually define sets of similar categories, e.g. learning a task module for mammals and learning another task module for different vehicles.
In order to quantitatively evaluate the performance of transplanting to dissimilar categories, we designed new task modules for additional testing. We considered the first four categories of the VOC dataset, i.e. aeroplane, bicycle, bird, and boat, to have dissimilar structures with mammals. In Experiment 2/Experiment 3, for the transplanting of a mammal category (let us take the cat for example) to a classification/segmentation module, we learned a “leave-one-out” classification/segmentation module to deal with all four dissimilar categories and all mammal categories except the cat.
Table 4 shows the performance of transplanting to a task module trained for both similar and dissimilar categories. Our method outperformed the baseline. Compared to the performance of transplanting to a task module oriented to similar categories in Table 3, transplanting to a more generic task modules for dissimilar categories hurt the segmentation performance but boosted the classification performance. It is because forcing a task module to handle dissimilar categories sometimes made the task module encode more generic and robust representations, while it may also let the task module ignore details of mammal categories.
5 Conclusions and discussion
In this paper, we focused on a new task, i.e. merging pre-trained teacher nets into a generic, modular transplant net with a few or even without training annotations. We discussed the importance and core challenges of this task. We developed the back-distillation algorithm as a theoretical solution to the challenging space-projection problem.
The back-distillation strategy significantly decreases the demand for training samples. Experimental results demonstrated the superior efficiency of our method. Our method without any training samples even outperformed the baseline with 100 training samples, as shown in Table 2(left).
The growth of a large transplant net for different categories and tasks can be divided into lots of elementary operations of network transplanting (see Fig. 1 and supplementary materials for more discussions). When the transplant net has been learned, we can optionally fine-tune task modules using training samples of multiple categories. Note that unlike the back distillation, the performance of fine-tuning depends on the number of training samples. Thus, given a few samples, whether an additional fine-tuning will increase or decrease the generality of the transplant net is a difficult question, and it requires sophisticated analysis in the future.
-  M. Andrychowicz, M. Denil, S. G. Colmenarejo, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. de Freitas. Learning to learn by gradient descent by gradient descent. In NIPS, 2016.
-  R. Anil, G. Pereyra, A. Passos, R. Ormandi, G. E. Dahl, and G. E. Hinton. Large scale distributed neural network training through online distillation. In ICLR, 2018.
-  X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS, 2016.
-  Y. Chen, M. W. Hoffman, S. G. Colmenarejo, M. Denil, T. P. Lillicrap, M. Botvinick, and N. de Freitas. Learning to learn without gradient descent by gradient descent. In ICML, 2017.
-  Y.-M. Chou, Y.-M. Chan, J.-H. Lee, C.-Y. Chiu, and C.-S. Chen. Unifying and merging well-trained deep neural networks for inference stage. In arXiv:1805.04980, 2018.
-  C. Fernando, D. Banarse, C. Blundell, Y. Zwols, D. Hay, A. A. Rusu, A. Pritzel, and D. Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. In arXiv:1701.08734, 2017.
-  N. Frosst and G. Hinton. Distilling a neural network into a soft decision tree. In arXiv:1711.09784, 2017.
Y. Ganin and V. Lempitsky.
Unsupervised domain adaptation in backpropagation.In ICML, 2015.
-  B. Hariharan, P. Arbelaez, L. Bourdev, S. Maji, and J. Malik. Semantic contours from inverse detectors. In ICCV, 2011.
-  I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. -vae: learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
-  G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. In NIPS Workshop, 2014.
-  P. Kontschieder, M. Fiterau, A. Criminisi, and S. R. Bulò. Deep neural decision forests. In ICCV, 2015.
-  K. Li and J. Malik. Learning to optimize. In arXiv:1606.01885, 2016.
-  C. Liu, B. Zoph, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy. Progressive neural architecture search. In arXiv:1712.00559, 2017.
-  J. Long, E. Shelhamer, and T. Darrel. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
-  A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive neural networks. In arXiv:1606.04671, 2016.
-  S. Sabour, N. Frosst, and G. E. Hinton. Dynamic routing between capsules. In NIPS, 2017.
-  R. Schwartz-Ziv and N. Tishby. Opening the black box of deep neural networks via information. In arXiv:1703.00810, 2017.
-  J. Schwarz, J. Luketina, W. M. Czarnecki, A. Grabska-Barwinska, Y. W. Teh, R. Pascanu, and R. Hadsell. Progress & compress: A scalable framework for continual learning. In arXiv:1805.06370, 2018.
-  N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In ICLR, 2017.
-  K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
-  S. Srinivas and F. Fleuret. Knowledge transfer with jacobian matching. In ICML, 2018.
-  R. Vuorio, D.-Y. Cho, D. Kim, and J. Kim. Meta continual learning. In arXiv:1806.06928, 2018.
-  N. Wolchover. New theory cracks open the black box of deep learning. In Quanta Magazine, 2017.
-  M. Wu, M. C. Hughes, S. Parbhoo, M. Zazzi, V. Roth, and F. Doshi-Velez. Beyond sparsity: Tree regularization of deep models for interpretability. In NIPS TIML Workshop, 2017.
-  T. Wu, X. Li, X. Song, W. Sun, L. Dong, and B. Li. Interpretable r-cnn. In arXiv:1711.05226, 2017.
-  J. Yoon, E. Yang, J. Lee, and S. J. Hwang. Lifelong learning with dynamically expandable networks. In ICLR, 2018.
-  J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In NIPS, 2014.
-  S. Zagoruyko and N. Komodakis. Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In arXiv:1612.03928, 2017.
Q. Zhang, Y. N. Wu, and S.-C. Zhu.
Interpretable convolutional neural networks.In CVPR, 2018.
-  Z. Zhong, J. Yan, and C.-L. Liu. Practical network blocks design with q-learning. In AAAI, 2018.
-  Z.-H. Zhou and J. Feng. Deep forest: Towards an alternative to deep neural networks. In IJCAI, 2017.
B. Zoph and Q. V. Le.
Neural architecture search with reinforcement learning.In ICLR, 2017.
-  B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. Learning transferable architectures for scalable image recognition. In arXiv:1707.07012, 2017.
Computation of gradients w.r.t. the distillation loss
In order to optimize the distillation loss, we need to compute , i.e. with respect to the following .
Thus, we first explore the close-form formulation of the function . As shown in the above equation, we can transform the back-propagation process for computing as a number of cascaded functions , which are derivatives of . Since we have formulated in the manuscript and it is easy to obtain , we mainly focus on the formulation of .
In general, it is not difficult to derive the derivative of any convolution operation. Here, we focus on the most common case, i.e. the convolution operation with a padding
and a stride of 1. Given a tensorand convolutional filter with weights and a bias term , the convolution can be written as . For VGG networks, people usually set . can further absorb by adding the channel to and adding the channel to with . Thus, we can obtain
where and . Thus, we can write the derivative of as
In this way, we obtain the close-form formulation of the function . We can easily compute
using the chain rule of back propagation. We can consider this process as aback-back-propagation.
About the case of using an enlarged
When we use an enlarged pseudo-gradient , we can obtain an enlarged gradient map . As discussed above, the computation of can be considered as a number of cascaded functions with the input , which are quite similar to the forward propagation in the neural network.
Visualization of network structures used in three experiments
How to learn a large transplant network
In the paper, we limit our attention to the core technique of back distillation for network transplanting and do preliminary experiments to demonstrate the effectiveness of the proposed method. Here, we would like to explain how to use the proposed back-distillation algorithm to gradually grow a large transplant network. The basic idea of growing a large transplant network has been shown in Figure 1 of the paper. We can divide the complex procedure of building a large transplant network into elementary operations of network transplanting.
Generally speaking, there are two types of operations during the learning of the transplant network, i.e. adding a new task module and adding a new category module.
Adding a new task module: Compared to adding category modules, adding task modules is relatively easier. There are two ways to add a task module.
Firstly, given a single or multiple pre-trained category modules and training samples of the category/categories, we can directly learn a new task module upon features of the category module(s). This learning process has been shown as “Step 1” in Figure 1, which follows traditional learning strategy without network transplanting.
Secondly, when the specific network is pre-trained with a single category module and multiple task modules, we can transplant the category module to a generic task module in the target transplant network. In this case, task modules (those do not correspond to ) will automatically be added to the transplant network, although these task modules only connect with a single category module . This case has been shown as “Step 2” in Figure 1, in which the Task-3 module is added to the transplant network when we connect the category- module to the Task-2 module of the transplant network.
Adding a new category module: We can also divide the insertion of a category modules into the following two cases. The first case is the traditional network transplanting that is introduced in this paper. The second case is to build more connections between category modules and task modules that have already been contained by the transplant network. This case is shown as “Step 3” in Figure 1. We learn new adapters to connect existing category modules and task modules via network transplanting.
Comparing our back-distillation with the Jacobian distillation
The loss for the Jacobian distillation  is similar to Equation (3). There are two essential differences between them, which make the Jacobian distillation not applicable to network transplanting. (i) The Jacobian distillation does not balance magnitudes of neural activations between the category module and the task module, which significantly increases the difficulties of network transplanting. (ii) More crucially, the Jacobian distillation uses real gradients of the network instead of using pseudo-gradients. However, when we fixed the upper task module during network transplanting, the task module usually blocks most signals during the forward propagation. Thus, during the back propagation, real gradients usually cannot pass ReLU layers in the task module to produce informative Jacobians for distillation.
The core challenge
To clarify the challenge of learning the adapter, we will compare the following two cases, i.e. 1) learning both the adapter and the task module and 2) learning the adapter but fixing the task module . The traditional method of directly optimizing the task loss can easily solve the first case but will be hampered in the second case.
Case 1, Learning both the adapter and the task module : This case corresponds to the traditional problem of deep learning. We initialize the task module
with random parameters, so in the first epoch of learning, input featureswill produce random activations in layers of both and , and positive neural activations will pass through ReLU layers to make random predictions . Successfully passing information to the final output is quite important, because we can obtain gradients of the task loss to optimize and . In this way, both and can be well learned.
Case 2, learning the adapter but fixing the task module : Unlike Case 1, we use a pre-trained task module , and we fix its parameters during the learning process. We only initialize the adapter with random parameters. However, as shown in Fig. 3(left) in the paper, the task module has vast forgotten space of its input feature, and can only pass very specific features to the final output. In other words, cannot pass most information of ’s features to the final output in early epochs. Thus, we will not obtain informative gradients to optimize parameters in .