Asymmetric Tri-training for Unsupervised Domain Adaptation

by   Kuniaki Saito, et al.

Deep-layered models trained on a large number of labeled samples boost the accuracy of many tasks. It is important to apply such models to different domains because collecting many labeled samples in various domains is expensive. In unsupervised domain adaptation, one needs to train a classifier that works well on a target domain when provided with labeled source samples and unlabeled target samples. Although many methods aim to match the distributions of source and target samples, simply matching the distribution cannot ensure accuracy on the target domain. To learn discriminative representations for the target domain, we assume that artificially labeling target samples can result in a good representation. Tri-training leverages three classifiers equally to give pseudo-labels to unlabeled samples, but the method does not assume labeling samples generated from a different domain.In this paper, we propose an asymmetric tri-training method for unsupervised domain adaptation, where we assign pseudo-labels to unlabeled samples and train neural networks as if they are true labels. In our work, we use three networks asymmetrically. By asymmetric, we mean that two networks are used to label unlabeled target samples and one network is trained by the samples to obtain target-discriminative representations. We evaluate our method on digit recognition and sentiment analysis datasets. Our proposed method achieves state-of-the-art performance on the benchmark digit recognition datasets of domain adaptation.


Deep Discriminative Learning for Unsupervised Domain Adaptation

The primary objective of domain adaptation methods is to transfer knowle...

Zero-Shot Domain Adaptation via Kernel Regression on the Grassmannian

Most visual recognition methods implicitly assume the data distribution ...

General Domain Adaptation Through Proportional Progressive Pseudo Labeling

Domain adaptation helps transfer the knowledge gained from a labeled sou...

A Fully Convolutional Tri-branch Network (FCTN) for Domain Adaptation

A domain adaptation method for urban scene segmentation is proposed in t...

Pseudo-Labeling Curriculum for Unsupervised Domain Adaptation

To learn target discriminative representations, using pseudo-labels is a...

Loss-based Sequential Learning for Active Domain Adaptation

Active domain adaptation (ADA) studies have mainly addressed query selec...

Structured Discriminative Tensor Dictionary Learning for Unsupervised Domain Adaptation

Unsupervised Domain Adaptation (UDA) addresses the problem of performanc...

1 Introduction

With the development of deep neural networks including deep convolutional neural networks (CNN)

(Krizhevsky et al., 2012)

, the recognition abilities of images and languages have improved dramatically. Training deep-layered networks with a large number of labeled samples enables us to correctly categorize samples in diverse domains. In addition, the transfer learning of CNN is utilized in many studies. For object detection or segmentation, we can transfer the knowledge of a CNN trained with a large-scale dataset by fine-tuning it on a relatively small dataset

(Girshick et al., 2014; Long et al., 2015a)

. Moreover, features from a CNN trained on ImageNet

(Deng et al., 2009) are useful for multimodal learning tasks including image captioning (Vinyals et al., 2015) and visual question answering (Antol et al., 2015).

One of the problems of neural networks is that although they perform well on the samples generated from the same distribution as the training samples, they may find it difficult to correctly recognize samples from different distributions at the test time. One example is images collected from the Internet, which may come in abundance and be fully labeled. They have a distribution different from the images taken from a camera. Thus, a classifier that performs well on various domains is important for practical use. To realize this, it is necessary to learn domain-invariantly discriminative representations. However, acquiring such representations is not easy because it is often difficult to collect a large number of labeled samples and because samples from different domains have domain-specific characteristics.

In unsupervised domain adaptation, we try to train a classifier that works well on a target domain on the condition that we are provided labeled source samples and unlabeled target samples during training. Most of the previous deep domain adaptation methods have been proposed mainly under the assumption that the adaptation can be realized by matching the distribution of features from different domains. These methods aimed to obtain domain-invariant features by minimizing the divergence between domains as well as a category loss on the source domain (Ganin & Lempitsky, 2014; Long et al., 2015b, 2016). However, as shown in (Ben-David et al., 2010), theoretically, if a classifier that works well on both the source and the target domains does not exist, we cannot expect a discriminative classifier for the target domain. That is, even if the distributions are matched on the non-discriminative representations, the classifier may not work well on the target domain. Since directly learning discriminative representations for the target domain, in the absence of target labels, is considered very difficult, we propose to assign pseudo-labels to target samples and train target-specific networks as if they were true labels.

Co-training and tri-training (Zhou & Li, 2005) leverage multiple classifiers to artificially label unlabeled samples and retrain the classifiers. However, the methods do not assume labeling samples from different domains. Since our goal is to classify unlabeled target samples that have different characteristics from labeled source samples, we propose asymmetric tri-training for unsupervised domain adaptation. By asymmetric, we mean that we assign different roles to three classifiers.

Figure 1: Outline of our model. We assign pseudo-labels to unlabeled target samples based on the predictions from two classifiers trained on source samples.

In this paper, we propose a novel tri-training method for unsupervised domain adaptation, where we assign pseudo-labels to unlabeled samples and train neural networks utilizing the samples. As described in Fig. 1, two networks are used to label unlabeled target samples and the remaining network is trained by the pseudo-labeled target samples. Our method does not need any special implementations. We evaluate our method on the digit classification task, traffic sign classification task and sentiment analysis task using the Amazon Review dataset, and demonstrate state-of-the-art performance in nearly all experiments. In particular, in the adaptation scenario, MNISTSVHN, our method outperformed other methods by more than 10%.

2 Related Work

As many methods have been proposed to tackle various tasks in domain adaptation, we present details of the research most closely related to our paper.

A number of previous methods attempted to realize adaptation by utilizing the measurement of divergence between different domains (Ganin & Lempitsky, 2014; Long et al., 2015b; Li et al., 2016). The methods are based on the theory proposed in (Ben-David et al., 2010), which states that the expected loss for a target domain is bounded by three terms: (i) expected loss for the source domain; (ii) domain divergence between source and target; and (iii) the minimum value of a shared expected loss. The shared expected loss means the sum of the loss on the source and target domain. As the third term, which is usually considered to be very low, cannot be evaluated when labeled target samples are absent, most methods try to minimize the first term and the second term. With regards to training deep architectures, the maximum mean discrepancy (MMD) or a loss of domain classifier network is utilized to measure the divergence corresponding to the second term (Gretton et al., 2012; Ganin & Lempitsky, 2014; Long et al., 2015b, 2016; Bousmalis et al., 2016). However, the third term is very important in training CNN, which simultaneously extract representations and recognize them. The third term can easily be large when the representations are not discriminative for the target domain. Therefore, we focus on how to learn target-discriminative representations considering the third term. In (Long et al., 2016) the focus was on the point we have stated and a target-specific classifier was constructed using a residual network structure. Different from their method, we constructed a target-specific network by providing artificially labeled target samples.

Several transductive methods use similarity of features to provide labels for unlabeled samples (Rohrbach et al., 2013; Khamis & Lampert, 2014). For unsupervised domain adaptation, in (Sener et al., 2016), a method was proposed to learn labeling metrics by using the

-nearest neighbors between unlabeled target samples and labeled source samples. In contrast to this method, our method explicitly and simply backpropagates the category loss for target samples based on pseudo-labeled samples. Our approach does not require any special modules.

Many methods proposed to give pseudo-labels to unlabeled samples by utilizing the predictions of a classifier and retraining it including the pseudo-labeled samples, which is called self-training. The underlying assumption of self-training is that one’s own high-confidence predictions are correct (Zhu, 2005). As the predictions are mostly correct, utilizing samples with high confidence will further improve the performance of the classifier. Co-training utilizes two classifiers, which have different views on one sample, to provide pseudo-labels (Blum & Mitchell, 1998; Tanha et al., 2011). Then, the unlabeled samples are added to training set if at least one classifier is confident about the predictions. The generalization ability of co-training is theoretically ensured (Balcan et al., 2004; Dasgupta et al., 2001) under some assumptions and applied to various tasks (Wan, 2009; Levin et al., 2003). In (Chen et al., 2011), the idea of co-training was incorporated into domain adaptation. Tri-training can be regarded as the extension of co-training (Zhou & Li, 2005). Similar to co-training, tri-training uses the output of three different classifiers to give pseudo-labels to unlabeled samples. Tri-training does not require partitioning features into different views; instead, tri-training initializes each classifier differently. However, tri-training does not assume that the unlabeled samples follow the different distributions from the ones which labeled samples are generated from. Therefore, we develop a tri-training method suitable for domain adaptation by using three classifiers asymmetrically.

In (Lee, 2013), the effect of pseudo-labels in a neural network was investigated. They argued that the effect of training a classifier with pseudo-labels is equivalent to entropy regularization, thus leading to a low-density separation between classes. In addition, in our experiment, we observe that target samples are separated in hidden features.

Figure 2: The proposed method includes a shared feature extractor (), classifiers for labeled samples (), which learn from labeled source samples, and newly labeled target samples. In addition, a target-specific classifier () learns from pseudo-labeled target samples. Our method first trains networks from only labeled source samples, then labels the target samples based on the output of . We train all architectures using them as if they are correctly labeled samples.

3 Method

In this section, we provide details of the proposed model for domain adaptation. We aim to construct a target-specific network by utilizing pseudo-labeled target samples. Simultaneously, we expect two labeling networks to acquire target-discriminative representations and gradually increase accuracy on the target domain.

We show our proposed network structure in Fig. 2. Here denotes the network which outputs shared features among three networks, and classify features generated from . Their predictions are utilized to give pseudo-labels. The classifier classifies features generated from , which is a target-specific network. Here learn from source and pseudo-labeled target samples and learns only from pseudo-labeled target samples. The shared network learns from all gradients from . Without such a shared network, another option for the network architecture we can think of is training three networks separately, but this is inefficient in terms of training and implementation. Furthermore, by building a shared network , and can also harness the target-discriminative representations learned by the feedback from .

The set of source samples is defined as , the unlabeled target set is , and the pseudo-labeled target set is .

3.1 Loss for Multiview Features Network

In the existing works (Chen et al., 2011) on co-training for domain adaptation, given features are divided into separate parts and considered to be different views.

As we aim to label target samples with high accuracy, we expect to classify samples based on different viewpoints. Therefore, we make a constraint for the weight of to make their inputs different to each other. We add the term to the cost function, where denote fully connected layers’ weights of and which are first applied to the feature . Each network will learn from different features with this constraint. The objective for learning is defined as



denotes the standard softmax cross-entropy loss function. We decided the trade-off parameter

based on validation split.

3.2 Learning Procedure and Labeling Method

Pseudo-labeled target samples will provide target-discriminative information to the network. However, since they certainly contain false labels, we have to pick up reliable pseudo-labels. Our labeling and learning method is aimed at realizing this.

The entire procedure of training the network is shown in Algorithm 1. First, we train the entire network with source training set . Here are optimized by Eq. (1) and is trained on standard category loss. After training on , to provide pseudo-labels, we use predictions of and , namely obtained from . When

denote the class which has the maximum predicted probability for

, we assign a pseudo-label to if the following two conditions are satisfied. First, we require to give pseudo-labels, which means two different classifiers agree with the prediction. The second requirement is that the maximizing probability of or exceeds the threshold parameter, which we set as 0.9 or 0.95 in the experiment. We suppose that unless one of two classifiers is confident of the prediction, the prediction is not reliable. If the two requirements are satisfied, is added to . To prevent the overfitting to pseudo-labels, we resample the candidate for labeling samples in each step. We set the number of the initial candidates as 5,000. We gradually increase the number of the candidates , where denotes the number of all target samples and denotes the number of steps, and we set the maximum number of pseudo-labeled candidates as 40,000. After the pseudo-labeled training set is composed, are updated by the objective Eq. (1) on the labeled training set . Then, are simply optimized by the category loss for .

Discriminative representations will be learned by constructing a target-specific network trained only on target samples. However, if only noisy pseudo-labeled samples are used for training, the network may not learn useful representations. Then, we use both source samples and pseudo-labeled samples for training to ensure the accuracy. Also, as the learning proceeds, will learn target-discriminative representations, resulting in an improvement in accuracy in . This cycle will gradually enhance the accuracy in the target domain.

  Input: data ,
  for  to  do
     Train with mini-batch from training set
  end for
   Labeling(,, )
  for  steps do
     for  to  do
        Train with mini-batch from training set
        Train with mini-batch from training set
     end for
      = Labeling(,, )
  end for
Algorithm 1 denotes the iteration of training. The function Labeling means the method of labeling. We assign pseudo-labels to samples when the predictions of and agree and at least one of them is confident of their predictions.

3.3 Batch Normalization for Domain Adaptation

Batch normalization (BN) (Ioffe & Szegedy, 2015), which whitens the output of the hidden layer in a CNN, is an effective technique to accelerate training speed and enhance the accuracy of the model. In addition, in domain adaptation, whitening the hidden layer’s output is effective for improving the performance, which make the distribution in different domains similar (Sun et al., 2016; Li et al., 2016).

Input samples of include both pseudo-labeled target samples and source samples. Introducing BN will be useful for matching the distribution and improves the performance. We add the BN layer in the last layer in .

4 Analysis

In this section, we provide a theoretical analysis to our approach. First, we provide an insight into existing theory, then we introduce a simple expansion of the theory related to our method.

In (Ben-David et al., 2010), an equation was introduced showing that the upper bound of the expected error in the target domain depends on three terms, which include the divergence between different domains and the error of an ideal joint hypothesis. The divergence between source and target domain, -distance, is defined as follows:

This distance is frequently used to measure the adaptability between different domains.

The ideal joint hypothesis is defined as , and its corresponding error is , where denotes the expected error on each hypothesis. The theorem is as follows.

Theorem 1.

(Ben-David et al., 2010)
Let be the hypothesis class. Given two different domains , we have

This theorem means that the expected error on the target domain is upper bounded by three terms, the expected error on the source domain, the domain divergence measured by the disagreement of the hypothesis, and the error of the ideal joint hypothesis. In the existing work (Ganin & Lempitsky, 2014; Long et al., 2015b), was disregarded because it was considered to be negligibly small. If we are provided with fixed features, we do not need to consider the term because the term is also fixed. However, if we assume that are obtained from the last fully connected layer of deep models, we note that is determined by the output of the layer, and further note the necessity of considering this term.

We consider the pseudo-labeled target samples set given false labels at the ratio of . The shared error of on is denoted as . Then, the following inequality holds:

We show a simple derivation of the inequality in the Supplementary material. In Theorem 1, we cannot measure in the absence of labeled target samples. We can approximately evaluate and minimize it by using pseudo-labels. Furthermore, when we consider the second term on the right-hand side, our method is expected to reduce this term. This term intuitively denotes the discrepancy between different domains in the disagreement of two classifiers. If we regard certain and as and , respectively, should be very low because training is based on the same labeled samples. Moreover, for the same reason, is expected to be low, although we use the training set instead of genuine labeled target samples. Thus, our method will consider both the second and the third term in Theorem 1.

                                            SOURCE MNIST SVHN MNIST SYN DIGITS SYN SIGNS
                                            TARGET MNIST-M MNIST SVHN SVHN GTSRB
Source Only w/o BN 59.1(56.6) 68.1(59.2) 37.2(30.5) 84.1(86.7) 79.2(79.0)
Source Only with BN 57.1 70.1 34.9 85.5 75.7
MMD (Long et al., 2015b) 76.9 71.1 - 88.0 91.1
DANN (Ganin & Lempitsky, 2014) 81.5 71.1 35.7 90.3 88.7
DRCN (Ghifary et al., 2016) - 82.0 40.1 - -
DSN (Bousmalis et al., 2016) 83.2 82.7 - 91.2 93.1
kNN-Ad (Sener et al., 2016) 86.7 78.8 40.3 - -
Ours w/o BN 85.3 79.8 39.8 93.1 96.2
Ours w/o weight constraint () 94.2 86.2 49.7 92.4 94.0
Ours 94.0 85.0 52.8 92.9 96.2
Table 1: Results of the visual domain adaptation experiment on digits and traffic signs dataset. In every setting, our method outperforms other method by a large margin. In source only results, we show the results reported in (Bousmalis et al., 2016) and (Ghifary et al., 2016) in parentheses.
MNISTMNIST-M: last pooling layer (a) Non-adapted (b) Adapted MNISTSVHN: last shared hidden layer (c) Non-adapted (d) Adapted
Figure 3: We confirm the effect our method by visualization of the learned representations by using -distributed stochastic neighbor embedding (t-SNE) (Maaten & Hinton, 2008). Red points are target samples and blue points are source samples. The samples are all from testing samples. (a), (c) The case where we only use source samples for training. (b), (d) The case of adaptation by our method. In both scenarios, MNISTSVHN and MNISTMNIST-M, we can see that the target samples are more dispersed through adaptation.

5 Experiment and Evaluation

We perform extensive evaluations of our method on image datasets and a sentiment analysis dataset. We evaluate the accuracy of target-specific networks in all experiments.

Visual Domain Adaptation For visual domain adaptation, we perform our evaluation on the digits datasets and traffic signs datasets. Digits datasets include MNIST (LeCun et al., 1998), MNIST-M (Ganin & Lempitsky, 2014), Street View House Numbers (SVHN) (Netzer et al., 2011), and Synthetic Digits (SYN DIGITS) (Ganin & Lempitsky, 2014). We further evaluate our method on traffic sign datasets including Synthetic Traffic Signs (SYN SIGNS) (Moiseev et al., 2013) and German Traffic Signs Recognition Benchmark (Stallkamp et al., 2011) (GTSRB). In total, five adaptation scenarios are evaluated in this experiment. As the datasets used for evaluation are varied in previous works, we extensively evaluate our method on the five scenarios.

We do not evaluate our method on Office (Saenko et al., 2010), which is the most commonly used dataset for visual domain adaptation. As pointed out by (Bousmalis et al., 2016), some labels in that dataset are noisy and some images contain other classes’ objects. Furthermore, many previous studies have evaluated the fine-tuning of pretrained networks using ImageNet. This protocol assumes the existence of another source domain. In our work, we want to evaluate the situation where we have access to only one source domain and one target domain.

Adaptation in Amazon Reviews To investigate the behavior on language datasets, we also evaluated our method on the Amazon Reviews dataset (Blitzer et al., 2006) with the same preprocessing as used by (Chen et al., 2011; Ganin et al., 2016). The dataset contains reviews on four types of products: books, DVDs, electronics, and kitchen appliances. We evaluate our method on 12 domain adaptation scenarios. The results are shown in Table 1.

Baseline Methods We compare our method with five methods for unsupervised domain adaptation including state-of-the art methods in visual domain adaptation; Maximum Mean Discrepancy (MMD) (Long et al., 2015b), Domain Adversarial Neural Network (DANN) (Ganin & Lempitsky, 2014), Deep Reconstruction Classification Network (DRCN) (Ghifary et al., 2016), Domain Separation Networks (DSN) (Bousmalis et al., 2016), and -Nearest Neighbor based adaptation (NN-Ad) (Sener et al., 2016). We cite the results of MMD from (Bousmalis et al., 2016)

. In addition, we compare our method with CNN trained only on source samples. We compare our method with Variational Fair AutoEncoder (VFAE)

(Louizos et al., 2015) and DANN (Ganin et al., 2016) in the Amazon Reviews experiment.

5.1 Implementation Detail

In experiments on image datasets, we employ the architecture of CNN used in (Ganin & Lempitsky, 2014). For a fair comparison, we separate the network at the hidden layer from which (Ganin & Lempitsky, 2014) constructed discriminator networks. Therefore, when considering one classifier, for example, , the architecture is identical to previous work. We also follow (Ganin & Lempitsky, 2014) in the other protocols. We set the threshold value for the labeling method as 0.95 in MNISTSVHN. In other scenarios, we set it as 0.9. We use MomentumSGD for optimization and set the momentum as , while the learning rate is determined on validation splits and uses either . is set 0.01 in all scenarios. In our Supplementary material, we provide details of the network architecture and hyper-parameters.

For experiments on the Amazon Reviews dataset, we use a similar architecture to that used in (Ganin et al., 2016): with sigmoid activated, one dense hidden layer with 50 hidden units, and softmax output. We extend the architecture to our method similarly in the architecture of CNN. is set as 0.001 based on the validation. Since the input is sparse, we use Adagrad (Duchi et al., 2011) for optimization. We repeat this evaluation 10 times and report mean accuracy.

(f) Comparision of accuracy of three network on SVHNMNIST
(g) -distance in MNISTMNIST-M
Figure 4: (a) (e): Comparison of the actual accuracy of pseudo-labels and learned network accuracy during training. The blue curve is the pseudo-label accuracy and the red curve is the learned network accuracy. Note that the labeling accuracy is computed using (the number of correctly labeled samples)/(the number of labeled samples). The green curve is the number of labeled target samples in each step. (f): Comparison of the accuracy of three networks in our model. Three networks almost simultaneously improve accuracy. (g): Comparison of the -distance of different methods. Our model slightly reduced the divergence of the domain compared with source-only trained CNN.

5.2 Experimental Result

In Tables 1 and 3, we show the main results of the experiments. When training only on source samples, the effect of the BN is not clear as in Tables 1. However, in all image recognition experiments, the effect of BN in our method is clear; at the same time, the effect of our method is also clear when we do not use BN in the network architecture. The effect of the weight constraint is obvious in MNISTSVHN.

MNISTMNIST-M First, we evaluate the adaptation scenario between the hand-written digits dataset MNIST and its transformed dataset MNIST-M. MNIST-M is composed by merging the clip of the background from BSDS500 datasets (Arbelaez et al., 2011). A patch is randomly taken from the images in BSDS500, merged to MNIST digits. Even with this simple domain shift, the adaptation performance of CNN is much worse than the case where it was trained on target samples. From 59,001 target training samples, we randomly select 1,000 labeled target samples as a validation split and tuned hyper-parameters.

Our method outperforms the other existing method by about 7%. Visualization of features in the last pooling layer is shown in Fig. 3LABEL:LABEL:. We can observe that the red target samples are more dispersed when adaptation is achieved. We show the comparison of the accuracy between the actual labeling accuracy on target samples during training and the test accuracy in Fig. 4. The test accuracy is very low at first, but as the steps increase, the accuracy becomes closer to that of the labeling accuracy. In this adaptation, we can clearly see that the actual labeling accuracy gradually improves with the accuracy of the network.

SVHNMNIST We increase the gap between distributions in this experiment. We evaluate adaptation between SVHN (Netzer et al., 2011) and MNIST in a ten-class classification problem. SVHN and MNIST have distinct appearance, thus this adaptation is a challenging scenario especially in MNIST

SVHN. SVHN is colored and some images contain multiple digits. Therefore, a classifier trained on SVHN is expected to perform well on MNIST, but the reverse is not true. MNIST does not include any samples containing multiple digits and most samples are centered in images, thus adaptation from MNIST to SVHN is rather difficult. In both settings, we use 1,000 labeled target samples to find the optimal hyperparameters.

We evaluate our method on both adaptation scenarios and achieved state-of-the-art performance on both datasets. In particular, for the adaptation MNISTSVHN, we outperformed other methods by more than 10%. In Fig. 3LABEL:LABEL:, we visualize the representations in MNISTSVHN. Although the distributions seem to be separated between domains, the red SVHN samples become more discriminative using our method compared with non-adapted embedding. We also show the comparison between actual labeling method accuracy and testing accuracy in Fig. 4LABEL:LABEL:. In this figure, we can see that the labeling accuracy rapidly drops in the initial adaptation stage. On the other hand, testing accuracy continues to improve, and finally exceeds the labeling accuracy. There are two questions about this interesting phenomenon. The first question is why does the labeling method continue to decrease despite the increase in the test accuracy? Target samples given pseudo-labels always include mistakenly labeled samples whereas those given no labels are ignored in our method. Therefore, the error will be reinforced in the target samples that are included in training set. The second question is why does the test accuracy continue to increase despite the lower labeling accuracy? The assumed reasons are that the network already acquires target discriminative representations in this phase and they can improve the accuracy using source samples and correctly labeled target samples.

In Fig. 4(f), we also show the comparison of accuracy of the three networks in SVHNMNIST. The accuracy of three networks is nearly the same in every step. The same thing is observed in other scenarios. From this result, we can state that the target-discriminative representations are shared in all three networks.

SYN DIGITSSVHN In this experiment, we aimed to address a common adaptation scenario from synthetic images to real images. The datasets of synthetic numbers (Ganin & Lempitsky, 2014) consist of 500,000 images generated from Windows fonts by varying the text, positioning, orientation, background and stroke colors, and the amount of blur. We use 479,400 source samples and 73,257 target samples for training, and 26,032 target samples for testing. We use 1,000 SVHN samples as a validation set.

Our method also outperforms other methods in this experiment. In this experiment, the effect of BN is not clear compared with other scenarios. The domain gap is considered small in this scenario as the performance of the source-only classifier shows. In Fig. 4(d), although the labeling accuracy is dropping, the accuracy of the learned network’s prediction is improving as in MNISTSVHN.

SYN SIGNSGTSRB This setting is similar to the previous setting, adaptation from synthetic images to real images, but we have a larger number of classes, namely 43 classes instead of 10. We use the SYN SIGNS dataset (Ganin & Lempitsky, 2014) for the source dataset and the GTSRB dataset (Stallkamp et al., 2011) for the target dataset, which consist of real traffic sign images. We select randomly 31,367 samples for target training samples and evaluate accuracy on the rest of the samples. A total of 3,000 labeled target samples are used for validation.

In this scenario, our method outperforms other methods. This result shows that our method is effective for the adaptation from synthesized images to real images, which have diverse classes. In Fig. 4(e), the same tendency as in MNISTSVHN is observed in this adaptation scenario.

Gradient stop branch None
MNISTMNIST-M 56.4 95.4 94.0
MNISTSVHN 47.7 47.5 52.8
SYN SIGNSGTRB 96.5 93.1 96.2
Table 2: Results of Gradient stop experiment. When stopping gradients from , we do not use backward gradients from to , and learns only from . When stopping gradients from , we do not use backward gradients from to , and learns from . None denotes our proposed method, we backward all gradients from all branches to . In these three adaptation scenarios, our method shows stable performance.

Gradient Stop Experiment We evaluate the effect of the target-specific network in our method. We stop the gradient from upper layer networks , and to examine the effect of . Table 2 shows three scenarios including the case where we stop the gradient from , and . In all scenarios, when we backward all gradients from , we obtain clear performance improvements.

In the experiment MNISTMNIST-M, we can assume that only the backpropagation from cannot construct discriminative representations for target samples and confirm the effect of . For the adaptation MNISTSVHN, the best performance is realized when receives all gradients from upper networks. Backwarding all gradients will ensure both target-specific discriminative representations in difficult adaptations. In SYN SIGNSGTSRB, backwarding only from produces the worst performance because these domains are similar and noisy pseudo-labeled target samples worsen the performance.

-distance From the theoretical results in (Ben-David et al., 2010),

-distance is usually used as a measure of domain discrepancy. The way of estimating empirical

-distance is simple, in which we train a classifier to classify a domain from each domains’ feature. Then, the approximate distance is calculated as , where is the generalization error of the classifier. In Fig. 4(g), we show the -distance calculated from each CNN features. We used linear SVM to calculate the distance. From this graph, we can see that our method certainly reduces the -distance compared with the CNN trained on only source samples. In addition, when comparing DANN and our method, although DANN reduces -distance much more than our method, our method shows superior performance. This indicates that minimizing the domain discrepancy is not necessarily an appropriate way to achieve better performance.

Amazon Reviews

Reviews are encoded in 5,000 dimensional vectors of bag-of-words unigrams and bigrams with binary labels. Negative labels are attached to the samples if they are ranked with 1–3 stars. Positive labels are attached if they are ranked with 4 or 5 stars. We have 2,000 labeled source samples and 2,000 unlabeled target samples for training, and between 3,000 and 6,000 samples for testing. We use 200 of labeled target samples for validation.

From the results in Table 3, our method performs better than VFAE (Louizos et al., 2015) and DANN (Ganin et al., 2016) in nine settings out of twelve. Our method is effective in learning a shallow network on different domains.

SourceTarget VFAE DANN Our method
booksdvd 79.9 78.4 80.7
bookselectronics 79.2 73.3 79.8
bookskitchen 81.6 77.9 82.5
dvdbooks 75.5 72.3 73.2
dvdelectronics 78.6 75.4 77.0
dvdkitchen 82.2 78.3 82.5
electronicsbooks 72.7 71.1 73.2
electronicsdvd 76.5 73.8 72.9
electronicskitchen 85.0 85.4 86.9
kitchenbooks 72.0 70.9 72.5
kitchendvd 73.3 74.0 74.9
kitchenelectronics 83.8 84.3 84.6
Table 3: Amazon Reviews experimental results. The accuracy (%) of the proposed method is shown with the result of VFAE (Louizos et al., 2015) and DANN (Ganin et al., 2016).

6 Conclusion

In this paper, we have proposed a novel asymmetric tri-training method for unsupervised domain adaptation, which is simply implemented. We aimed to learn discriminative representations by utilizing pseudo-labels assigned to unlabeled target samples. We utilized three classifiers, two networks assign pseudo-labels to unlabeled target samples and the remaining network learns from them. We evaluated our method both on domain adaptation on a visual recognition task and a sentiment analysis task, outperforming other methods. In particular, our method outperformed the other methods by more than 10% in the MNISTSVHN adaptation task.

7 Acknowledgement

This work was funded by ImPACT Program of Council for Science, Technology and Innovation (Cabinet Office, Government of Japan) and supported by CREST, JST.


  • Antol et al. (2015) Antol, Stanislaw, Agrawal, Aishwarya, Lu, Jiasen, Mitchell, Margaret, Batra, Dhruv, Lawrence Zitnick, C., and Parikh, Devi. Vqa: Visual question answering. In ICCV, 2015.
  • Arbelaez et al. (2011) Arbelaez, Pablo, Maire, Michael, Fowlkes, Charless, and Malik, Jitendra. Contour detection and hierarchical image segmentation. PAMI, 33(5):898–916, 2011.
  • Balcan et al. (2004) Balcan, Maria-Florina, Blum, Avrim, and Yang, Ke. Co-training and expansion: Towards bridging theory and practice. In NIPS, 2004.
  • Ben-David et al. (2010) Ben-David, Shai, Blitzer, John, Crammer, Koby, Kulesza, Alex, Pereira, Fernando, and Vaughan, Jennifer Wortman. A theory of learning from different domains. Machine learning, 79(1-2):151–175, 2010.
  • Blitzer et al. (2006) Blitzer, John, McDonald, Ryan, and Pereira, Fernando. Domain adaptation with structural correspondence learning. In EMNLP, 2006.
  • Blum & Mitchell (1998) Blum, Avrim and Mitchell, Tom. Combining labeled and unlabeled data with co-training. In COLT, 1998.
  • Bousmalis et al. (2016) Bousmalis, Konstantinos, Trigeorgis, George, Silberman, Nathan, Krishnan, Dilip, and Erhan, Dumitru. Domain separation networks. In NIPS, 2016.
  • Chen et al. (2011) Chen, Minmin, Weinberger, Kilian Q, and Blitzer, John. Co-training for domain adaptation. In NIPS, 2011.
  • Dasgupta et al. (2001) Dasgupta, Sanjoy, Littman, Michael L, and McAllester, David. Pac generalization bounds for co-training. In NIPS, 2001.
  • Deng et al. (2009) Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Li, Kai, and Fei-Fei, Li. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
  • Duchi et al. (2011) Duchi, John, Hazan, Elad, and Singer, Yoram. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12(7):2121–2159, 2011.
  • Ganin & Lempitsky (2014) Ganin, Yaroslav and Lempitsky, Victor. Unsupervised domain adaptation by backpropagation. In ICML, 2014.
  • Ganin et al. (2016) Ganin, Yaroslav, Ustinova, Evgeniya, Ajakan, Hana, Germain, Pascal, Larochelle, Hugo, Laviolette, François, Marchand, Mario, and Lempitsky, Victor. Domain-adversarial training of neural networks. JMLR, 17(59):1–35, 2016.
  • Ghifary et al. (2016) Ghifary, Muhammad, Kleijn, W Bastiaan, Zhang, Mengjie, Balduzzi, David, and Li, Wen. Deep reconstruction-classification networks for unsupervised domain adaptation. In ECCV, 2016.
  • Girshick et al. (2014) Girshick, Ross, Donahue, Jeff, Darrell, Trevor, and Malik, Jitendra. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
  • Gretton et al. (2012) Gretton, Arthur, Borgwardt, Karsten M, Rasch, Malte J, Schölkopf, Bernhard, and Smola, Alexander. A kernel two-sample test. JMLR, 13(3):723–773, 2012.
  • Ioffe & Szegedy (2015) Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167, 2015.
  • Khamis & Lampert (2014) Khamis, Sameh and Lampert, Christoph H. Coconut: Co-classification with output space regularization. In BMVC, 2014.
  • Krizhevsky et al. (2012) Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
  • LeCun et al. (1998) LeCun, Yann, Bottou, Léon, Bengio, Yoshua, and Haffner, Patrick. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  • Lee (2013) Lee, Dong-Hyun.

    Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.

    In ICML workshop on Challenges in Representation Learning, 2013.
  • Levin et al. (2003) Levin, Anat, Viola, Paul A, and Freund, Yoav. Unsupervised improvement of visual detectors using co-training. In ICCV, 2003.
  • Li et al. (2016) Li, Yanghao, Wang, Naiyan, Shi, Jianping, Liu, Jiaying, and Hou, Xiaodi. Revisiting batch normalization for practical domain adaptation. arXiv:1603.04779, 2016.
  • Long et al. (2015a) Long, Jonathan, Shelhamer, Evan, and Darrell, Trevor. Fully convolutional networks for semantic segmentation. In CVPR, 2015a.
  • Long et al. (2015b) Long, Mingsheng, Cao, Yue, Wang, Jianmin, and Jordan, Michael I. Learning transferable features with deep adaptation networks. In ICML, 2015b.
  • Long et al. (2016) Long, Mingsheng, Zhu, Han, Wang, Jianmin, and Jordan, Michael I. Unsupervised domain adaptation with residual transfer networks. In NIPS, 2016.
  • Louizos et al. (2015) Louizos, Christos, Swersky, Kevin, Li, Yujia, Welling, Max, and Zemel, Richard. The variational fair autoencoder. arXiv:1511.00830, 2015.
  • Maaten & Hinton (2008) Maaten, Laurens van der and Hinton, Geoffrey. Visualizing data using t-sne. JMLR, 9(11):2579–2605, 2008.
  • Moiseev et al. (2013) Moiseev, Boris, Konev, Artem, Chigorin, Alexander, and Konushin, Anton. Evaluation of traffic sign recognition methods trained on synthetically generated data. In ACIVS, 2013.
  • Netzer et al. (2011) Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng, Andrew Y. Reading digits in natural images with unsupervised feature learning. In

    NIPS workshop on deep learning and unsupervised feature learning

    , 2011.
  • Rohrbach et al. (2013) Rohrbach, Marcus, Ebert, Sandra, and Schiele, Bernt. Transfer learning in a transductive setting. In NIPS, 2013.
  • Saenko et al. (2010) Saenko, Kate, Kulis, Brian, Fritz, Mario, and Darrell, Trevor. Adapting visual category models to new domains. In ECCV, 2010.
  • Sener et al. (2016) Sener, Ozan, Song, Hyun Oh, Saxena, Ashutosh, and Savarese, Silvio. Learning transferrable representations for unsupervised domain adaptation. In NIPS, 2016.
  • Stallkamp et al. (2011) Stallkamp, Johannes, Schlipsing, Marc, Salmen, Jan, and Igel, Christian. The german traffic sign recognition benchmark: a multi-class classification competition. In IJCNN, 2011.
  • Sun et al. (2016) Sun, Baochen, Feng, Jiashi, and Saenko, Kate. Return of frustratingly easy domain adaptation. In AAAI, 2016.
  • Tanha et al. (2011) Tanha, Jafar, van Someren, Maarten, and Afsarmanesh, Hamideh. Ensemble based co-training. In BNAIC, 2011.
  • Vinyals et al. (2015) Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. In CVPR, 2015.
  • Wan (2009) Wan, Xiaojun. Co-training for cross-lingual sentiment classification. In ACL, 2009.
  • Zhou & Li (2005) Zhou, Zhi-Hua and Li, Ming. Tri-training: Exploiting unlabeled data using three classifiers. TKDE, 17(11):1529–1541, 2005.
  • Zhu (2005) Zhu, Xiaojin. Semi-supervised learning literature survey. Technical report, University of Wisconsin-Madison, 2005.

Proof of Theorem

We introduce the derivation of theorem of the main paper. The ideal joint hypothesis is defined as , and its corresponding error is , where denotes the expected error on each hypothesis.

We consider the pseudo-labeled target samples set given false labels at the ratio of . The minimum shared error on is denoted as . Then, the following inequality holds:


The probabiliy of false labels in the pseudo-labeled set is . When we consider 0-1 loss function for , the difference between the error based on the true labeled set and pseudo-labeled set is

Then, the difference in the expected error is,

From the characteritic of the loss function, the triangle inequality will hold, then

From this result, the main inequality holds. ∎

CNN Architectures and training detail

Four types of architectures are used for our method, which is based on (Ganin & Lempitsky, 2014). The network topology is shown in Figs 6, 7 and 8. The other hyperparameters are decided on the validation splits. The learning rate is set to 0.05 in SVHNMNIST. In the other scenarios, it is set to 0.01. The batchsize for training is set as 128, the batchsize for training is set as 64 in all scenarios.

In MNISTMNIST-M, the dropout rate used in the experiment is 0.2 for training , 0.5 for training . In MNISTSVHN, we did not use dropout. We decreased learning rate to 0.001 after step 10. In SVHNMNIST, the dropout rate used in the experiment is 0.5. In SYNDIGITSSVHN, the dropout rate used in the experiment is 0.5. In SYNSIGNSGTSRB, the dropout rate used in the experiment is 0.5.

Supplementary experiments on MNISTMnist-M

We observe the behavior of our model when increasing the number of steps up to one hundred. We show the result in Fig. 5. Our model’s accuracy gets about 97%. In our main experiments, we set the number of steps thirty, but from this experiment, further improvements can be expected.

Figure 5: The behavior of our model when increasing the number of steps up to 100. Our model achieves accuracy of about 97%.
Figure 6: The architecture used for MNISTMNIST-M. We added BN layer in the last convolution layer and FC layers in . We also used dropout in our experiment.
Figure 7: The architecture used for training SVHN. In MNISTSVHN, we added a BN layer in the last FC layer in . In SVHNMNIST, SYN DigitsSVHN, we added BN layer in the last convolution layer in and FC layers in , and also used dropout.
Figure 8: The architecture used in the adaptation Synthetic SignsGTSRB. We added a BN layer after the last convolution layer in and also used dropout.