1 Introduction
Many machine learning algorithms assume that the training and test data are independent and identically distributed (i.i.d.). However, this assumption rarely holds in practice as the data is likely to change over time and space. Even though stateoftheart Deep Convolutional Neural Network features are invariant to low level cues to some degree
[2, 3, 4], Donahue et al. [5]showed that they still are susceptible to domain shift. Instead of collecting labelled data and training a new classifier for every possible scenario, unsupervised domain adaptation methods
[6, 7, 8, 9, 10, 1] try to compensate for the degradation in performance by transferring knowledge from labelled source domains to unlabelled target domains. A recently proposed CORAL method [1] aligns the secondorder statistics of the source and target distributions with a linear transformation. Even though it is “frustratingly easy”, it works well for unsupervised domain adaptation. However, it relies on a linear transformation and is not endtoend: it needs to first extract features, apply the transformation, and then train an SVM classifier in a separate step.In this work, we extend CORAL to incorporate it directly into deep networks by constructing a differentiable loss function that minimizes the difference between source and target correlations–the CORAL loss. Compared to CORAL, our proposed Deep CORAL approach learns a nonlinear transformation that is more powerful and also works seamlessly with deep CNNs. We evaluate our method on standard benchmark datasets and show stateoftheart performance.
2 Related Work
Previous techniques for unsupervised adaptation consisted of reweighting the training point losses to more closely reflect those in the test distribution [11, 12] or finding a transformation in a lowerdimensional manifold that brings the source and target subspaces closer together. Reweighting based approaches often assume a restricted form of domain shift–selection bias–and are thus not applicable to more general scenarios. Geodesic methods [13, 7] bridge the source and target domain by projecting source and target onto points along a geodesic path [13], or finding a closedform linear map that transforms source points to target [7]. [14, 8] align the subspaces by computing the linear map that minimizes the Frobenius norm of the difference between the top eigenvectors. In contrast, CORAL [1] minimizes domain shift by aligning the secondorder statistics of source and target distributions.
Adaptive deep neural networks have recently been explored for unsupervised adaptation. DLID [15] trains a joint source and target CNN architecture with two adaptation layers. DDC [16] applies a single linear kernel to one layer to minimize Maximum Mean Discrepancy (MMD) while DAN [17] minimizes MMD with multiple kernels applied to multiple layers. ReverseGrad [18] adds a binary classifier to explicitly confuse the two domains.
Our proposed Deep CORAL approach is similar to DDC, DAN, and ReverseGrad in the sense that a new loss (CORAL loss) is added to minimize the difference in learned feature covariances across domains, which is similar to minimizing MMD with a polynomial kernel. However, it is more powerful than DDC (which aligns sample means only), much simpler to optimize than DAN and ReverseGrad, and can be integrated into different layers or architectures seamlessly.
3 Deep CORAL
We address the unsupervised domain adaptation scenario where there are no labelled training data in the target domain, and propose to leverage both the deep features pretrained on a large generic domain (such as Imagenet
[19]) and the labelled source data. In the meantime, we also want the final learned features to work well on the target domain. The first goal can be achieved by initializing the network parameters from the generic pretrained network and finetuning it on the labelled source data. For the second goal, we propose to minimize the difference in secondorder statistics between the source and target feature activations, i.e. the CORAL loss. Figure 1 shows a sample Deep CORAL architecture using our proposed correlation alignment layer for deep domain adaptation. We refer to Deep CORAL as any deep network incorporating the CORAL loss for domain adaptation.3.1 CORAL Loss
We first describe the CORAL loss between two domains for a single feature layer. Suppose we are given sourcedomain training examples , with labels , , and unlabeled target data , . Suppose the number of source and target data are and respectively. Here both and are the dimensional deep layer activations of input that we are trying to learn. Suppose () indicates the th dimension of the th source (target) data example and () denote the feature covariance matrices.
We define the CORAL loss as the distance between the secondorder statistics (covariances) of the source and target features:
(1) 
where denotes the squared matrix Frobenius norm. The covariance matrices of the source and target data are given by:
(2) 
(3) 
where 1
is a column vector with all elements equal to 1.
The gradient with respect to the input features can be calculated using the chain rule:
(4) 
(5) 
We use batch covariances and the network parameters are shared between two networks.
3.2 Endtoend Domain Adaptation with CORAL Loss
We describe our method by taking a multiclass classification problem as the running example. As mentioned before, the final deep features need to be both discriminative enough to train strong classifier and invariant to the difference between source and target domains. Minimizing the classification loss itself is likely to lead to overfitting to the source domain, causing reduced performance on the target domain. On the other hand, minimizing the CORAL loss alone might lead to degenerated features. For example, the network could project all of the source and target data to a single point, making the CORAL loss trivially zero. However, no strong classifier can be constructed on these features. Joint training with both the classification loss and CORAL loss is likely to learn features that work well on the target domain:
(6) 
where denotes the number of CORAL loss layers in a deep network and is a weight that trades off the adaptation with classification accuracy on the source domain. As we show below, these two losses play counterparts and reach an equilibrium at the end of training, where the final features are expected to work well on the target domain.
4 Experiments
We evaluate our method on a standard domain adaptation benchmark – the Office dataset [6]. The Office dataset contains 31 object categories from an office environment in 3 image domains: , , and .
We follow the standard protocol of [7, 17, 5, 16, 18] and use all the labelled source data and all the target data without labels. Since there are 3 domains, we conduct experiments on all 6 shifts, taking one domain as the source and another as the target.
In this experiment, we apply the CORAL loss to the last classification layer as it is the most general case–most deep classifier architectures (e.g., convolutional, recurrent) contain a fully connected layer for classification. Applying the CORAL loss to other layers or other network architectures should be straightforward.
The dimension of last fully connected layer () was set to the number of categories (31) and initialized with . The learning rate of is set to 10 times the other layers as it was training from scratch. We initialized the other layers with the parameters pretrained on ImageNet [19] and kept the original layerwise parameter settings. In the training phase, we set the batch size to 128, base learning rate to , weight decay to , and momentum to 0.9. The weight of the CORAL loss (
) is set in such way that at the end of training the classification loss and CORAL loss are roughly the same. It seems be a reasonable choice as we want to have a feature representation that is both discriminative and also minimizes the distance between the source and target domains. We used Caffe
[21] and BVLC Reference CaffeNet for all of our experiments.We compare to 7 recently published methods: CNN [20] (no adaptation), GFK [7], SA [8], TCA [22], CORAL [1], DDC [16], DAN [17]. GFK, SA, and TCA are manifold based methods that project the source and target distributions into a lowerdimensional manifold and are not endtoend deep methods. DDC adds a domain confusion loss to AlexNet and finetunes it on both the source and target domain. DAN is similar to DDC but utilizes a multikernel selection method for better mean embedding matching and adapts in multiple layers. For direct comparison, DAN in this paper uses the hidden layer . For GFK, SA, TCA, and CORAL, we use the feature finetuned on the source domain ( in [1]) as it achieves better performance than generic pretrained features, and train a linear SVM [8, 1]. To have a fair comparison, we use accuracies reported by other authors with exactly the same setting or conduct experiments using the source code provided by the authors.
From Table 1 we can see that Deep CORAL (DCORAL) achieves better average performance than CORAL and the other 6 baseline methods. In three 3 out of 6 shifts, it achieves the highest accuracy. For the other 3 shifts, the margin between DCORAL and the best baseline method is very small ().
AD  AW  DA  DW  WA  WD  AVG  
GFK  52.40.0  54.70.0  43.20.0  92.10.0  41.80.0  96.20.0  63.4 
SA  50.60.0  47.40.0  39.50.0  89.10.0  37.60.0  93.80.0  59.7 
TCA  46.80.0  45.50.0  36.40.0  81.10.0  39.50.0  92.20.0  56.9 
CORAL  65.70.0  64.30.0  48.50.0  96.10.0  48.20.0  99.80.0  70.4 
CNN  63.80.5  61.60.5  51.10.6  95.40.3  49.80.4  99.00.2  70.1 
DDC  64.40.3  61.80.4  52.10.8  95.00.5  52.20.4  98.50.4  70.6 
DAN  65.80.4  63.80.4  52.80.4  94.60.5  51.90.5  98.80.6  71.3 
DCORAL  66.80.6  66.40.4  52.80.2  95.70.3  51.50.3  99.20.1  72.1 
To get a better understanding of Deep CORAL, we generate three plots for domain shift AW. In Figure 2(a) we show the training (source) and testing (target) accuracies for training with vs. without CORAL loss. We can clearly see that adding the CORAL loss helps achieve much better performance on the target domain while maintaining strong classification accuracy on the source domain.
In Figure 2(b) we visualize both the classification loss and the CORAL loss for training w/ CORAL loss. As the last fully connected layer is randomly initialized with , in the beginning the CORAL loss is very small while the classification loss is very large. After training for a few hundred iterations, these two losses are about the same. In Figure 2(c) we show the CORAL distance between the domains for training w/o CORAL loss (setting the weight to 0). We can see that the distance is getting much larger ( times larger compared to training w/ CORAL loss). Comparing Figure 2(b) and Figure 2(c), we can see that even though the CORAL loss is not always decreasing during training, if we set its weight to 0, the distance between source and target domains becomes much larger. This is reasonable as finetuning without domain adaptation is likely to overfit the features to the source domain. Our CORAL loss constrains the distance between source and target domain during the finetuning process and helps to maintain an equilibrium where the final features work well on the target domain.
5 Conclusion
In this work, we extended CORAL, a simple yet effective unsupervised domain adaptation method, to perform endtoend adaptation in deep neural networks. Experiments on standard benchmark datasets show stateoftheart performance. Deep CORAL works seamlessly with deep networks and can be easily integrated into different layers or network architectures.
References
 [1] Sun, B., Feng, J., Saenko, K.: Return of frustratingly easy domain adaptation. In: AAAI. (2016)
 [2] Peng, X., Sun, B., Ali, K., Saenko, K.: What do deep cnns learn about objects? In: ICLR Workshop Track. (2015)
 [3] Peng, X., Sun, B., Ali, K., Saenko, K.: Learning deep object detectors from 3d models. In: ICCV. (2015)
 [4] Sun, B., Peng, X., Saenko, K.: Generating large scale image datasets from 3d cad models. In: CVPR’15 Workshop on The Future of Datasets in Vision. (2015)
 [5] Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., Darrell, T.: Decaf: A deep convolutional activation feature for generic visual recognition. In: ICML. (2014)
 [6] Saenko, K., Kulis, B., Fritz, M., Darrell, T.: Adapting visual category models to new domains. In: ECCV. (2010)
 [7] Gong, B., Shi, Y., Sha, F., Grauman, K.: Geodesic flow kernel for unsupervised domain adaptation. In: CVPR. (2012)
 [8] Fernando, B., Habrard, A., Sebban, M., Tuytelaars, T.: Unsupervised visual domain adaptation using subspace alignment. In: ICCV. (2013)
 [9] Sun, B., Saenko, K.: From virtual to reality: Fast adaptation of virtual object detectors to real domains. In: BMVC. (2014)
 [10] Sun, B., Saenko, K.: Subspace distribution alignment for unsupervised domain adaptation. In: BMVC. (2015)
 [11] Jiang, J., Zhai, C.: Instance Weighting for Domain Adaptation in NLP. In: ACL. (2007)
 [12] Huang, J., Smola, A.J., Gretton, A., Borgwardt, K.M., Schölkopf, B.: Correcting sample selection bias by unlabeled data. In: NIPS. (2006)
 [13] Gopalan, R., Li, R., Chellappa, R.: Domain adaptation for object recognition: An unsupervised approach. In: ICCV. (2011)
 [14] Harel, M., Mannor, S.: Learning from multiple outlooks. In: ICML. (2011)

[15]
Chopra, S., Balakrishnan, S., Gopalan, R.:
Dlid: Deep learning for domain adaptation by interpolating between domains.
In: ICML Workshop. (2013)  [16] Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T.: Deep domain confusion: Maximizing for domain invariance. CoRR abs/1412.3474 (2014)
 [17] Long, M., Cao, Y., Wang, J., Jordan, M.I.: Learning transferable features with deep adaptation networks. In: ICML. (2015)

[18]
Ganin, Y., Lempitsky, V.:
Unsupervised domain adaptation by backpropagation.
In: ICML. (2015)  [19] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., FeiFei, L.: Imagenet: A largescale hierarchical image database. In: CVPR. (2009)
 [20] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS. (2012)
 [21] Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 (2014)
 [22] Pan, S.J., Tsang, I.W., Kwok, J.T., Yang, Q.: Domain adaptation via transfer component analysis. In: IJCAI. (2009)