The official repository (in PyTorch) for the ECCV 2020 paper "Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference".
Multi-task networks are commonly utilized to alleviate the need for a large number of highly specialized single-task networks. However, two common challenges in developing multi-task models are often overlooked in literature. First, enabling the model to be inherently incremental, continuously incorporating information from new tasks without forgetting the previously learned ones (incremental learning). Second, eliminating adverse interactions amongst tasks, which has been shown to significantly degrade the single-task performance in a multi-task setup (task interference). In this paper, we show that both can be achieved simply by reparameterizing the convolutions of standard neural network architectures into a non-trainable shared part (filter bank) and task-specific parts (modulators), where each modulator has a fraction of the filter bank parameters. Thus, our reparameterization enables the model to learn new tasks without adversely affecting the performance of existing ones. The results of our ablation study attest the efficacy of the proposed reparameterization. Moreover, our method achieves state-of-the-art on two challenging multi-task learning benchmarks, PASCAL-Context and NYUD, and also demonstrates superior incremental learning capability as compared to its close competitors.READ FULL TEXT VIEW PDF
The official repository (in PyTorch) for the ECCV 2020 paper "Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference".
, and monocular depth estimation[12, 27]. Typically, these tasks are handled by CNNs independently, i.e., a separate model is optimized for each task, resulting in several task-specific models (Fig. (a)a). However, real-world problems are more complex and require models to perform multiple tasks on-demand without significantly compromising each task’s performance. For example, an interactive advertisement system tasked with displaying targeted content to its audience should be able to detect the presence of humans in its viewpoint effectively, estimate their gender and age group, recognize their head pose, etc. At the same time, there is a need for flexible models able to gradually add more tasks to their knowledge, without forgetting previously known tasks or having to re-train the whole model from scratch. For instance, a car originally deployed with lane and pedestrian detection functionalities can be extended with depth estimation capabilities post-production.
When it comes to learning multiple tasks under a single model, multi-task learning (MTL) techniques [2, 54] have been employed in the literature. On the one hand, encoder-focused approaches [41, 25, 36, 10, 43, 33, 1, 61] emphasize learning feature representations from multi-task supervisory signals by employing architectures that encode shared and task-specific information. On the other hand, decoder-focused approaches [63, 65, 66, 62] utilize the multi-task feature representations learned at the encoding stage to distill cross-task information at the decoding stage, thus refining the original feature representations. In both cases, however, the joint learning from multiple supervisory signals (i.e., tasks) can hinder the individual task performance if the associated tasks point to conflicting gradient directions during the update step of the shared feature representations (Fig. (b)b). Formally this is known as task interference or negative transfer and has been well documented in the literature [25, 39, 69]. To suppress negative transfer, several approaches [6, 21, 59, 16, 69, 56, 39]
dynamically re-weight each task’s loss function or re-order the task learning, to find a ‘sweet spot’ where individual task performance does not degrade significantly. Arguably, such approaches mainly focus on mitigating the negative transfer problem in the MTL architectures above, rather than eliminating it (see Section3.2). At the same time, existing works seem to disregard the fact that MTL models are commonly desired to be incremental, i.e., information from new tasks should be continuously incorporated while existing task knowledge is preserved. In existing works, the MTL model has to be re-trained from scratch if the task dictionary changes; this is arguably sub-optimal.
Recently, task-conditional networks  emerged as an alternative for MTL, inspired by work in multi-domain learning [49, 50]. That is, performing separate forward passes within an MTL model, one for each task, every time activating a set of task-specific residual responses on top of the shared responses. Note that, this is useful for many real-world setups (e.g., an MTL model deployed in a mobile phone with limited resources that adapts its responses according to the task at hand), and particularly for incremental learning (e.g., a scenario where the low-level tasks should be learned before the high-level ones). However, the proposed architecture in  is prone to task interference due to the inherent presence of shared modules, which is why the authors introduced an adversarial learning scheme on the gradients to minimize the performance degradation. Moreover, the model needs to be trained from scratch if the task dictionary changes.
All given, existing works primarily focus on either improving the multi-task performance or reducing the number of parameters and computations in the MTL model. In this paper, we take a different route and explicitly tackle the problems of incremental learning and task interference in MTL. We show that both problems can be addressed simply by reparameterizing the convolutional operations of a neural network. In particular, building upon the task-conditional MTL direction, we propose to decompose each convolution into a shared part that acts as a filter bank encoding common knowledge, and task-specific modulators that adapt this common knowledge uniquely for each task. Fig. (c)c illustrates our approach, RCM (Reparameterized Convolutions for Multi-task learning). Unlike existing works, the shared part in our case is not trainable to explicitly avoid negative transfer. Most notably, as any number of task-specific modulators can be introduced in each convolution, our model can incrementally solve more tasks without interfering with the previously learned ones. Our results demonstrate that the proposed RCM can outperform state-of-the-art methods in multi-task (Section 4.6) and incremental learning (Section 4.7) experiments. At the same time, we address the common multi-task challenge of task interference by construction, by ensuring tasks can only update task-specific components and cannot interact with each other.
Multi-task learning (MTL) aims at developing models that can solve a multitude of tasks [2, 54]. In computer vision, MTL approaches can roughly be divided into encoder-focused and decoder-focused ones. Encoder-focused approaches primarily emphasize on architectures that can encode multi-purpose feature representations through supervision from multiple tasks. Such encoding is typically achieved, for example, via feature fusion , branching [25, 43, 36, 61], self-supervision , attention , or filter grouping . Decoder-focused approaches start from the feature representations learned at the encoding stage, and further refine them at the decoding stage by distilling information across tasks in a one-off , sequential , recursive , or even multi-scale  manner. Due to the inherent layer sharing, the approaches above typically suffer from task interference. Several works proposed to dynamically re-weight the loss function of each task [6, 21, 59, 56], sort the order of task learning , or adapt the feature sharing between ‘related’ and ‘unrelated’ tasks , to mitigate the effect of negative transfer. In general, existing MTL approaches have primarily focused on improving multi-task performance or reducing the network parameters and computations. Instead, in this paper, we look at the largely unexplored problems of incremental learning and negative transfer in MTL models and propose a principled way to tackle them.
Incremental learning (IL) is a paradigm that attempts to augment the existing knowledge by learning from new data. IL is often used, for example, when aiming to add new classes  to an existing model, or learn new domains . It aims to mitigate ‘catastrophic forgetting’ , the phenomenon of forgetting old tasks as new ones are learned. To minimize the loss of existing knowledge, Li and Hoiem  optimized the new task while preserving the old task’s responses. Other works [23, 29] constrained the optimization process to minimize the effect learning has on weights important for older tasks. Rebuffi et al.  utilized exemplars that best approximate the mean of the learned classes in the feature space to preserve performance. Note that the performance of such techniques is commonly upper bounded by the joint training of all tasks. More relevant to our work, in a multi-domain setting, a few approaches [49, 50, 53, 37] utilize a pre-trained network that remains untouched and instead learn domain-specific components that adapt the behavior of the network to address the performance drop common in IL techniques. Inspired by this research direction, we investigate the training of parts of the network, while keeping the remaining components constant from initialization amongst all tasks. This technique not only addresses catastrophic forgetting but also task interference, which is crucial in MTL.
of filters and tensors within CNNs has been explored in the literature. In particular, filter-wise decomposition into a product of low-rank filters, filter groups , a basis of filter groups , etc. have been utilized. In contrast, tensor-wise examples include SVD decomposition [9, 64], CP-decomposition , Tucker decomposition , Tensor-Train decomposition , Tensor-Ring decomposition , T-Basis , etc. These techniques have been successfully used for compressing neural networks or reducing their inference time. Instead, in this paper, we utilize decomposition differently. We decompose each convolutional operation into two components: a shared and a task-specific part. Note that although we utilize the SVD decomposition for simplicity, the same principles hold for other decomposition types too.
In this section, we present techniques to adapt a CNN architecture, such that it can increasingly learn new tasks in an MTL setting while scaling more efficiently than simply adding single-task models. Section 3.1 introduces the problem formulation. Section 3.2 demonstrates the effect of task interference in MTL and motivates the importance of CNN reparameterization. Section 3.3 presents techniques to reparameterize CNNs and limit the parameter increase with respect to task-specific models.
Given tasks and input tensor , we aim to learn a function that holds for task , where and are the shared and task-specific parameters respectively. Unlike existing approaches [36, 41] which learn such functions on the layer level of the network, i.e., explicitly designing shared and task-specific layers, we aim to learn on a block-level by reparameterizing the convolutional operation, and adapting its behaviour conditioned on the task , as depicted in Fig. (b)b and Fig. (c)c. By doing so, we can explicitly address the task interference and catastrophic forgetting problems within an MTL setting.
To motivate the importance of addressing task interference by construction, we analyze the task-specific gradient directions on the shared modules of a state-of-the-art MTL model. Specifically, we utilize the work of , who used a discriminator to enforce indistinguishable gradients amongst tasks.
We acquire the gradients from the training dataset of PASCAL-Context  for each task, using minibatches of size 128, yielding 40 minibatches. We then use the Representation Similarity Analysis (RSA), proposed in 
for transfer learning, as a means to quantify the correlation of the gradients amongst the different tasks. Fig.3 depicts the task gradient correlations at different depths of a ResNet-26 model , trained to have indistinguishable gradients in the output layer . It can be seen that there is a limited gradient correlation amongst the tasks, demonstrating that addressing task interference indirectly (here with the use of adversarial learning on the gradients) is a very challenging problem. We instead follow a different direction and propose to utilize reparameterizations with shared components amongst different tasks that are untouched during the training process, and each task being able to optimize only its parameters. As such, task interference is eliminated by construction.
We define a convolutional operation for the single-task learning setup, Fig. (a)a. denotes the parameters of a single convolutional layer (we omit the bias to simplify notation) for a kernel size and channels. is the input tensor volume at a given spatial location ( and
are expressed in vector notation), andis the scalar response. Assuming such filters, the convolutional operator can be rewritten in matrix notation as , where provides responses, and . In a single-task setup:
where and are the task-specific parameters and responses for a given convolutional layer, respectively. The total number of parameters for this setup is . Our goal is to reparameterize in Eqn. 1 as:
using a set of shared () and task-specific () parameters for each convolutional layer of the backbone. Our formulation aims to retain the prediction performance of the original convolutional layer (Eq. 1), while simultaneously reducing the rate in which the total number of parameters grows. The complexity now becomes , which is less than for standard layers. We argue that this reparameterization is necessary for coping with task interference and incremental learning in an MTL setup, in which we only optimize for task-specific parameters , while keeping the shared parameters intact. Note that, when adding a new task , we do not need to train the entire network from scratch as in . We only optimize for each layer of the reparameterized CNN.
We denote our reparameterized convolutional layer as a matrix multiplication between the two sets of parameters: . In order to find a set of parameters that approximates the single-task weights a natural choice is to minimize the Frobenius norm directly.
Even though direct minimization of this metric is appealing due to its simplicity, it poses some major caveats.
(i) It assumes that all directions in the parameter space affect the final performance for task in the same way and are thus penalized uniformly. However, two different solutions for with the same Frobenius norm can yield drastically different losses.
(ii) This approximation is performed independently for each convolutional layer, neglecting the chain effect an inaccurate prediction in one layer can have in the succeeding layers. In the remainder of this section, we propose different techniques to address these limitations.
We implement the Reparameterized Convolution (RC) as a stack of two 2D convolutional layers without non-linearity in between, with having a spatial filter size and being a convolution (Fig. (b)b)222 To ensure compliance with ImageNet initialization, the new architecture is first pre-trained on ImageNet using the publicly available training script from PyTorch
To ensure compliance with ImageNet
initialization, the new architecture is first pre-trained on ImageNet using the publicly available training script from PyTorch.. We optimize only
directly on the task-specific loss function using stochastic gradient descent while keeping the shared weightsconstant. This ensures that training for one task is independent of other tasks, ruling out interference amongst tasks while optimizing the metric of interest.
Normalized Feature Fusion. One can view , a row in matrix , as a soft filter adaptation mechanism, i.e., a modulator which generates new task-specific filters from a given filter bank , depicted in Fig. (b)b. However, instead of training the vector directly, we propose its reparameterization into two terms, a vector term , and a scalar term as:
where denotes the Euclidean norm. We refer to this reparameterization as Normalized Feature Fusion (NFF), depicted in Fig. (c)c. NFF provides an easier optimization process in comparison to an unconstrained . This reparametrization enforces to be unit length and point in the direction which best merges the filter bank. The vector norm learns independently the appropriate scale of the newly generated filters, and thus the scale of the activation. Directly optimizing attempts to learn both jointly, which is a harder optimization problem. Normalizing weight tensors has been generally explored for speeding up the convergence of the optimization process [7, 55, 60]. In our work, we use it differently and demonstrate empirically (see Section 4.5) that such a reparameterization in series with a filter bank also improves performance in the MTL setting. As seen in Eq. 3, additional learnable parameters are introduced in the training process ( and ), however, can be computed after training and used directly for deployment, eliminating additional overhead.
Response Initialization. We build upon the findings of matrix/tensor decomposition literature [9, 64] that network weights/responses lie on a low dimensional subspace. We further assume that such a subspace can be beneficial for multiple tasks, and thus good for network initialization under a MTL setup. To this end, we identify a meaningful subspace of the responses for the generation of a better filter bank when compared to that directly learned by pre-training on ImageNet. More formally, let be the responses for input tensor , where are the pre-trained ImageNet weights. We define as a matrix containing responses of with the mean vector subtracted. We compute the eigen-decomposition of the covariance matrix
(using Singular Value Decomposition, SVD), where
is a diagonal matrix of the corresponding eigenvalues. We can now initialize the shared convolution parameterswith , and the task-specific with . We refer to this initialization methodology as Response Initialization (RI). We point the reader to the supplementary material for more details.
We focus our evaluation on dense prediction tasks, making use of two datasets. We conduct the majority of the experiments on PASCAL , and more specifically, PASCAL-Context . We address edge detection (Edge), semantic segmentation (SemSeg), human parts segmentation (Parts), surface normals estimation (Normals), and saliency (Sal). We evaluate single-task performance using optimal dataset F-measure (odsF)  for edge detection, mean intersection over union (mIoU) for semantic segmentation, human parts and saliency, and finally mean error (mErr) for surface normals. Labels for human parts segmentation are acquired from , while for saliency and surface normals from .
We further evaluate the proposed method on the smaller NYUD dataset 
, comprised of indoor scenes, on edge detection (Edge), semantic segmentation (SemSeg), surface normals estimation (Normals), and depth (Depth). The evaluation metrics for edge detection, semantic segmentation, and surface normals estimation are identical to those for PASCAL-Context, while for depth we use root mean squared error (RMSE).
All of our experiments make use of the DeepLabv3+ architecture , originally designed for semantic segmentation, which performs competitively for all tasks of interest as demonstrated in . DeepLabv3+ encodes multi-scale contextual information by utilizing a ResNet  encoder with a-trous convolutions  and an a-trous spatial pyramid pooling (ASPP) module, while a decoder with a skip connection refines the predictions. Unless otherwise stated, we use a ResNet-18 (R-18) based DeepLabv3+, and report the mean performance of five runs for each experiment333Baseline comparisons to competing methods, as well as additional backbone experiments, can be found in the supplementary material..
where is either 1 or 0 if a lower or a greater value indicates better performance, respectively, for a performance measure . P indicates the total number of tasks.
We investigate the level of task-specific adaptation required for a common backbone to perform competitively to single-task models, while additionally eliminating negative transfer. In other words, the necessity for task-specific modules, i.e., convolutions (Convs) and batch normalizations (BNs). Specifically, we optimize for task-specific Convs, BNs, or both along the network’s depth. Modules that are not being optimized maintain their ImageNet pre-trained parameters. Table 1 presents the effect on performance, while Fig. 4 depicts the total number of parameters with respect to the number of tasks. Experiments vary from common Convs and BNs (Freeze encoder) to task-specific Convs and BNs (Single-task), and anything in-between.
The model utilizing a common backbone pre-trained on ImageNet (Freeze encoder), as expected, is unable to perform competitively to the single-task counterpart, with a performance drop of 14.98%. Task-specific BNs significantly improve performance with a percentage drop of 5.76%, at a minimal increase in parameters (Fig. 4). The optimization of Convs is essential for competitive performance to single-task, with a percentage drop of 0.62%. However, the increase in parameters is comparable to single-task, which is undesirable (Fig. 4).
To validate the proposed methodology from Section 3, we conduct an ablation study, presented in Table 2. We additionally report the performance of a model trained jointly on all tasks, consisting of a fully shared encoder and task-specific decoders (Multi-task). This multi-task model is not trained in an IL setup but merely serves as a reference to the traditional multi-tasking techniques. We report a performance drop of 3.32% with respect to the single-task setup.
Reparameterized Convolution. We first develop a new baseline for our proposed reparameterization, where we replace every convolution with the RC (Section 3.3) counterpart. As seen in Table 2, RC achieves a performance drop of 2.13%, outperforming the 3.32% drop of the multi-task baseline, as well as the Task-specific BNs (Table 1) that achieved a performance drop of 5.76%. This observation corroborates the claim made in Section 4.4 that task-specific adaptation of the convolutions is essential for a model to perform competitively for all tasks. Additionally, we demonstrate that even without training entirely task-specific convolutions, as in Table 1 (Task-specific Convs), a performance boost can still be observed at a smaller magnitude, while the total number of parameters scales at a slower rate (Fig. 4). RCM in Fig. 4 depicts the parameter scaling of all the RC-based methods introduced in Table 2, described in this section. As such, improvements in performance from this baseline do not stem from an increase in network capacity.
Response Initialization. We investigate the effect on the performance of a more meaningful filter bank, RI (Section 3.3), against the filter bank learned by directly pre-training the RC architecture on ImageNet. In Table 2 we report the performance of our proposed model when directly pre-trained on ImageNet (Table 2-RC), and with the RI based filter bank (Table 2-RC+RI). Compared to the RC model, the performance significantly improves from a 2.13% drop to a 1.12% drop with the RC+RI model. This observation clearly demonstrates that the filter bank generated using our proposed RI approach is beneficial for better weight initialization.
Normalized Feature Fusion. We replace the unconstrained task-specific components of RC with the proposed NFF (Section 3.3). We demonstrate in Table 2 that NFF improves the performance no matter the initialization of the filter bank. RC improves from a 2.13% drop to a 2.07% in RC+NFF, while RC+RI improved from a 1.12% drop to 0.99% for RC+RI+NFF.
The architecture used for the remaining experiments is the Reparameterized Convolution (RC) with Normalized Feature Fusion (NFF), initialized using the Response Initialization (RI) methodology. This architecture is denoted as RCM.
|ASTMT (R-18 w/o SE) ||71.20||64.31||57.79||15.06||66.59||3.49|
|ASTMT (R-26 w SE) ||71.00||64.61||57.25||15.00||64.70||4.12|
|Series RA ||70.62||65.99||55.32||14.27||66.08||2.97|
|Parallel RA ||70.84||66.51||56.56||14.16||66.36||2.09|
|ASTMT (R-18 w/o SE) ||68.60||30.69||23.94||0.60||6.96|
|ASTMT (R-26 w SE) ||73.50||30.07||24.32||0.63||7.56|
|Series RA ||67.56||31.87||23.35||0.60||5.88|
|Parallel RA ||68.02||32.13||23.20||0.59||5.02|
In this work, we focus on comparing to task-conditional methods that can address MTL. We compare the performance of our method to Series Residual Adapter (Series RA)  and Parallel RA . Series and Parallel RAs learn multiple visual domains by optimizing domain-specific residual adaptation modules (rather than using RCM as in our work, Fig. (c)c) on an ImageNet pre-trained backbone. Since both methods were developed for multi-domain settings, we optimize them using our own pipeline, ensuring a fair comparison amongst the methods while additionally benchmarking the capabilities of multi-domain methods in a multi-task setup. We further report the performance of ASTMT , which utilizes an architecture resembling that of Parallel RA  with Squeeze-and-Excitation (SE) blocks  and adversarial task disentanglement of gradients. Specifically, we report the performance of the models using a ResNet-26 (R-26) DeepLab-V3+ with SE as reported in , and also optimize with the use of their codebase a ResNet-18 model without SE. The latter model uses an architecture resembling more closely that of the other methods since SE can be additionally incorporated in the others as well. We report the average performance drop with respect to our single-task baseline.
The results for PASCAL-Context (Table 3) and NYUD (Table 4) demonstrate that our method achieves the best performance, outperforming the other methods that make use of RA modules. This demonstrates that although the RA can perform competitively in multi-domain settings, placing the convolution in series without non-linearity is a more promising direction for the drastic adaptations required for different tasks in a multi-task learning setup.
We visualize in Fig. 5 the learned representations of single-task, Parallel RA , and RCM across tasks and network depths. For each task and layer combination, we compute a common PCA basis for the methods above and depict the first three principal components as RGB values. For all tasks and layers of the network, the representations of RCM closely resemble those of the single-task models. Simultaneously, Parallel RA is unable to adapt the convolution behavior to the extent required to be comparable to single-task models.
|ASTMT (R-18 w/o SE) ||70.70||14.84||55.32||50.49||64.34||11.77|
|Series RA ||70.62||14.27||65.99||55.32||66.08||2.83|
|Parallel RA ||70.84||14.16||66.51||56.56||66.36||1.73|
|ASTMT (R-18 w/o SE) ||63.91||57.33||68.67||14.12||64.43||3.76|
|Series RA ||65.99||55.32||70.62||14.27||66.08||2.39|
|Parallel RA ||66.51||56.56||70.84||14.16||66.36||1.88|
We further evaluate the methods from Section 4.6 in the incremental learning (IL) setup. In other words, we investigate the capabilities of the models to learn new tasks without the need to be completely retrained on the entire task dictionary. We divide the tasks of PASCAL-Context into three groups, (i) edge detection and surface normals (low-level tasks), (ii) saliency (mid-level task) and (iii) semantic segmentation and human parts segmentation (high-level tasks). IL experiments are conducted by allowing the base network to initially use knowledge from either (i) or (iii), and reporting the capability for the optimized model to learn additional tasks without affecting the performance of the already learned tasks (the performance drop is calculated over the new tasks that were not used in the initial training). In the IL setup, ASTMT  is initially trained using an R-18 backbone without SE (a comparable backbone to the competing methods for a fair comparison) on the subset of the tasks (either i or iii). New tasks can be incorporated by training their task-specific modules independently. On the other hand, Series RA, Parallel RA, and RCM, were designed to be inherently incremental due to directly optimizing only the task-specific modules. Consequently, their task-specific performance in the IL setup is identical to that reported in Section 4.6.
In Tables 5 and 6 we report the performance of tasks that are utilized to generate the initial knowledge of the model in grey (important for ASTMT ), while in black the performance of the incrementally learned tasks. As shown in both tables, and in particular Table 5, ASTMT does not perform competitively in the IL experiments. This observation further demonstrates the importance of utilizing generic filter banks that can be adapted based on the task-specific needs, in particular for IL setups. We consider research in generic multi-task filter banks to be a promising direction.
We have presented a novel method of a convolutional operation reparameterization and its application to training multi-task learning architectures. These reparameterized architectures can be applied on a multitude of different tasks, and allow the CNN to be inherently incremental, while additionally eliminating task interference, all by construction. We evaluate our model on two datasets and multiple tasks, and show experimentally that it outperforms competing baselines that address similar challenges. We further demonstrate its efficacy when compared to the state-of-the-art task-conditional multi-task method, which is unable to tackle incremental learning.
Acknowledgments. This work was sponsored by Advertima AG and co-financed by Innosuisse. We thank the anonymous reviewers for their valuable feedback.
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference
Menelaos Kanakis David Bruggemann Suman Saha
Stamatios Georgoulis Anton Obukhov Luc Van Gool
We based our implementation details on the work of , listed below for completeness.
Generic hyperparamaters. All models are optimized using SGD with a learning rate 0.005, momentum 0.9, weight decay 0.0001, and the “poly” learning rate schedule 
. We use a single GPU with a minibatch of 8 images. The input images during training are augmented with random horizontal flips and random scaling in the range [0.5, 2.0] in 0.25 increments. The validity of these hyperparameters has already been tested in, and hence they are used in all our experiments too, in order to ensure fair comparisons amongst different methods.
Dataset specific hyperparameters. PASCAL-Context 
models are trained for 60 epochs. The spatial size of the input images is 512512. NYUD  models are trained for 200 epochs. The spatial size of the input images is 425
560. Images of insufficient size are padded with the mean color.
Task weighting and loss functions. As is common in multi-task learning (MTL), losses require careful loss weighting [39, 62, 21, 56], where each loss is task-dependent. For edge detection, we optimize the binary cross-entropy (BCE) loss, scaled by 50. Due to the class imbalance between the edge and non-edge pixels, edge pixels are penalized with a weight 0.95, while non-edge pixels with a scale of 0.05, accommodating [24, 38]. For evaluation, we set the maximum allowed mislocalization of the optimal dataset F-measure (odsF)  to 0.0075 and 0.011 for PASCAL-Context and NYUD, respectively, using the package of . Semantic segmentation and human parts segmentation are optimized with cross-entropy loss, weighted by the factors of 1 and 2, respectively. Predictions of surface normals (normalized to unit vectors) and depth modalities are penalized using the loss, scaled by 10 and 1, respectively. Saliency is optimized using the BCE loss, weighted by a factor of 5.
In Section 3.3 of the main text (Response initialization, RI), we introduced the methodology for the generation of a better filter bank when compared to that directly learned by pre-training on ImageNet, and demonstrated improved performance when utilizing RI in Section 4. In this section, we present additional detail.
Recall that we defined the responses of a convolutional layer for an input tensor , where are the pre-trained ImageNet weights. We specify as a matrix containing responses of with the mean vector subtracted. To generate the new filter bank, we first compute the eigen-decomposition of the covariance matrix (using Singular Value Decomposition, SVD), where is an orthogonal matrix with the eigenvectors on the columns, and is a diagonal matrix of the corresponding eigenvalues. We can now utilize which acts as a method to project to () and from () a latent space. Thus, we can rewrite , with the centering operation being of importance due to the space being generated from centred responses. This gives rise to
where , initialized by , represents the task-specific parameters optimized independently for each task , and is implemented as a convolution. The non-trainable shared parameters are defined as and implemented as a convolution, with being the filter size of . The bias can be added to the running mean of the batchnorm following the convolution .
To ensure our re-implementation provides a stable baseline, Table 7 compares the single-task performance of our implementation using a ResNet-18 based DeepLabv3+, the results from  using a ResNet-18 based FPN , and the results from  who utilized a ResNet-26 based DeepLabv3+. We demonstrate that our single-task baseline outperforms both works on every task, and even though the numbers are not directly comparable due to minor implementation differences, it provides a verification of a strong baseline.
We additionally compare the proposed RCM (Reparameterized Convolutions for Multi-task learning) with respect to the single-task performance on the DeepLabv3+ with the deeper ResNet34 (R-34)  backbone. Results for PASCAL-Context  and NYUD  can be seen in Table 8 and Table 9, respectively. As seen, the percentage drops of and for PASCAL-Context and NYUD respectively are comparable to that of the ResNet18 backbone reported in the main paper.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1971–1978. Cited by: §4.1.
Pushing the boundaries of boundary detection using deep learning. arXiv preprint arXiv:1511.07386. Cited by: Appendix 0.A.
Overcoming catastrophic forgetting by incremental moment matching. In Advances in neural information processing systems, pp. 4652–4662. Cited by: §2.
Fast scene understanding for autonomous driving. arXiv preprint arXiv:1708.02550. Cited by: §1, §2.
Icarl: incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001–2010. Cited by: §2.
International Conference on Computational Learning Theory, pp. 545–560. Cited by: §3.3.
A modulation module for multi-task learning with applications in image retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 401–416. Cited by: §1, §2.