It is well known that the success of machine learning methods on visual recognition tasks is highly dependent on access to large labeled datasets. Unfortunately, model performance often drops significantly on data from a new deployment domain, a problem known as dataset shift, or dataset bias . Changes in the visual domain can include lighting, camera pose and background variation, as well as general changes in how the image data is collected. While this problem has been studied extensively in the domain adaptation literature , progress has been limited by the lack of large-scale challenge benchmarks.
Challenge competitions drive innovation in computer vision, however, most challenges build their train and test splits by randomly sampling data from the same domain. An assumption of test data having same distribution as train data rarely holds in real-life deployment scenarios. Cross-domain benchmarks[28, 36, 3, 45] have been created for evaluating domain adaptation algorithms, but they often have low task diversity, small domain shifts and small dataset size.
This paper introduces the Visual Domain Adaptation (VisDA) dataset and a challenge of the same name to encourage further progress in developing more robust domain transfer methods. The challenge task is to train a model on a “source” domain and then update it so that its performance improves on a new “target” domain, without using any target annotations. Our challenge includes two tracks, illustrated in Figure 1. The first track introduces the more traditional adaptation for object classification task. The second one suggests to tackle the relatively less-studied semantic segmentation task, which involves labeling each pixel of an input image with a semantic label.
The VisDA2017 challenge focuses on the domain shift from simulated to real imagery–a challenging shift that has many practical applications in computer vision. This type of “synthetic-to-real” domain shift is important in many real-world situations when labeled imagery is difficult or expensive to collect (autonomous decision making in robotics, medical imaging, etc.), whereas synthetic rendering pipeline can produce virtually infinite amounts of labeled data ones set up. For this reason we generated the largest cross-domain synthetic-to-real object classification dataset to date with over 280K images in the combined training, validation and testing sets. For the semantic segmentation track we augmented existing datasets for a total of approximately 30k images across three domains.
The VisDA2017 challenge focuses on unsupervised
domain adaptation (UDA) for which the deployment domain images are not labeled. For each task, we provide labeled samples from training domain (source) and unlabeled samples from validation (target) and a different test (target) domains. While there may be scenarios where target labels are available (so called “supervised domain adaptation”), we focus on purely unsupervised adaptation case as it is more challenging and often more realistic. Many unsupervised domain adaptation evaluation protocols use some labeled target domain data to select hyperparameters. However, assuming labeled target data goes against the UDA problem statement. For this reason, we collect twodifferent target domains, one for validation of hyperparameters and one for testing the models.
In this paper, we present both our baseline results and most interesting results obtained by participants during the actual challenge. In addition to our multi-domain datasets released during the main timeline of the challenge, we plan to fully opensource and support our model collection along with scripts and metadata required for proper rendering, and propose a more challenging version of the classification dataset as well as present multiple more challenging experiment setups - all in order to lay the foundation for future research on this topic.
2 Related Work
There has been a lot of prior work on visual domain adaptation, ranging from the earlier shallow feature methods [36, 2, 9] to the more recent deep adaptation approaches [12, 47]. A review of existing work in this area is beyond the scope of this paper; we refer the reader to a recent comprehensive survey . Several benchmark datasets have been collected and used to evaluate visual domain adaptation, most notable are summarized in Table 1. Majority of popular benchmarks lack task diversity: the most common cross-domain datasets focus on the image classification task, i.e. digits of different styles, objects  or faces  under varying conditions. Other tasks such as detection , structure prediction [34, 7, 33] and sequence labeling  have been relatively overlooked.
Classification Datasets. Adaptation of image classification methods has been among the most extensively studied problems of visual domain adaptation. One of the difficulties in re-using existing datasets to create multi-domain benchmarks is that the same set of categories must be shared among all domains. For digits (ten categories, 0-9), the most popular benchmark setup consists of three domains: MNIST (handwritten digits) , USPS (handwritten digits)  and SVHN (street view house numbers) . Digit images are sometimes synthetically augmented to create additional domains, for example by inverting colors or using randomly chosen backgrounds .
The Office dataset  is a popular benchmark for real-world objects. It contains 31 object categories captured in three domains: office environment images taken with a high quality camera (DSLR), the same environment captured with a low quality webcam (WEBCAM), and images downloaded from the amazon.com website (AMAZON).
Besides the small size of these benchmarks, another problem is the relatively small domain shifts, such as the shift between two different sensors (DSLR vs Webcam in the Office dataset ), or between two very similar handwritten digit datasets (MNIST vs USPS). Over time, improvements in underlying image representations and adaptation techniques have closed the domain gap on these benchmarks, and more challenging domain shifts are now needed to drive further progress. Another issue is the small scale. Modern computer vision methods require a lot of training data, while cross-domain datasets such as Office Dataset  only contain several hundred of images. The Cross-Dataset Testbed 
is a more recent classification benchmark. its “dense” version contains 40 classes extracted from Caltech256, Bing, SUN, and Imagenet with a minimum of 20 images per class in each dataset. It is significantly larger than Office, however, some domains are fairly close as they were collected in a similar way from web search engines. On the Caltech-Imagenet shift, adaptation performance has reached close to 90% accuracy.
The popularity of the image classification task as a testbed may be due to the relative simplicity of the task and the lower effort required to engineer a good baseline model. Compared with other vision problems such as object detection, activity detection in video, or structured prediction, image classification is simpler and less computationally expensive to explore. Moreover, many state-of-the-art classification models are readily available for use as a baseline upon which adaptation can be applied. At the same time, other tasks have characteristics that could present unique challenges for domain adaptation. In this work, we propose experimental setups for both the more common classification task, and the less studied semantic segmentation task.
Semantic Segmentation Datasets. Semantic segmentation methods assign object labels to each pixel of an input image. Dataset annotation is a highly labor-intensive process, this is why there are only very few semantic segmentation datasets designed specifically for domain adaptation. Two datasets that are frequently paired together for visual segmentation tasks are SYNTHIA  and CityScapes . SYNTHIA provides a collection of synthetically generated urban images that mimic car dash-cam footage, while CityScapes provides images of real urban street scenes. Other synthetic street-view datasets include GTA5  and Virtual KITTI . Our VisDA benchmark combines GTA5, CityScapes, and data from the recently released Berkeley Deep Drive/Nexar dataset .
Synthetic Datasets. Synthetic data augmentation has been extensively employed in computer vision research. More specifically, 3D models have been utilized to generate synthetic images with variable object poses, textures, and backgrounds . Recent usage of 3D simulation has been extended to multiple vision tasks such as object detection [30, 41]
, pose estimation, robotic simulation , semantic segmentation . Popular 3D model databases of common objects that may be used in visual domain adaptation tasks include ObjectNet3D , ShapeNet and the related ModelNet . Table 2 compares the VisDA classification dataset to existing synthetic object datasets.
3 VisDA-C: Classification Dataset
The VisDA Classification (VisDA-C) dataset provides a large-scale testbed for studying unsupervised domain adaptation in image classification. The dataset contains three splits (domains), each with the same 12 object categories:
training domain (source): synthetic renderings of 3D models from different angles and with different lighting conditions,
validation domain (target): a real-image domain consisting of images cropped from the Microsoft COCO dataset ,
testing domain (target): a real-image domain consisting of images cropped from the Youtube Bounding Box dataset 
We use different target domains for the validation and test splits to prevent hyper-parameter tuning on the test data. Unsupervised domain adaptation is usually done in a transductive manner, meaning that unlabeled test data is actively used to train the model. However, it is not possible to tune hyper-parameter on the test data, since it has no labels. Despite this fact, the lack of established validation sets often leads to poor experimental protocols where the labeled test set is used for this purpose. In our benchmark, we provide a validation set to mimic the more realistic deployment scenario where the target domain is unknown at training time and test labels are not available for hyper-parameter tuning. This setup also discourages algorithms that are designed to handle a specific target domain. It is important to mention that the validation and test sets are different domains, so over-tuning to one can potentially degrade performance on another.
3.1 Dataset Acquisition
Training Domain: CAD-Synthetic Images. The synthetic dataset was generated by rendering 3D models of the same object categories as in the real data from different angles and under different lighting conditions. As shown in Table 8, we obtained 1,907 models in total and generated 152,397 synthetic images. We used four main sources of models that are indicated with a sec prefix of the corresponding image filename. These four sources include manually chosen subsets of ShapenetCore , NTU 3D , SHREC 2010  with some labels retrieved from TSB  and our own collection of 3D CAD models from 3D Warehouse SketchUp.
We used twenty different camera yaw and pitch combinations with four different light directions per model. The lighting setup consists of ambient and sun light sources in 1:3 proportion. Objects were rotated, scaled and translated to match the floor plane, duplicate faces and vertices were removed, and the camera was automatically positioned to capture the entire object with a margin around it. For textured models, we also rendered their un-textured versions with a plain grey albedo. In total, we generated 152,397 synthetic images to form the synthetic source domain. Per-category image numbers are shown in Table 8 (supplementary), and Figure 2 shows samples of training domain data.
Validation Domain: MS COCO. The validation dataset for the classification track is built from the Microsoft COCO  Training and Validation
splits. In total, the MS COCO dataset contains 174,011 images. We used annotations provided by the COCO dataset to find and crop relevant object in each image. All images were padded by retaining an additional ~50% of its cropped height and width (i.e. by dividing the height and width by). Padded image patches whose height or width was under 70 pixels were excluded to avoid extreme image transformations on later stages. In total, we collected 55,388 object images that fall into the chosen twelve categories. We took all images from each of twelve categories with the exception of the “person” category, which was reduced to 4,000 images in order to balance the overall number of images per category (the original “person” category has more than 120k images). See Figure 2 for sample validation domain data. The breakdown of the validation domain dataset by the number of images per category is shown in Table 8 (supplementary).
Testing Domain: YouTube Bounding Boxes. Due to the overlap in object category labels with the other two domains, we chose the YouTube Bounding Boxes (YT-BB) dataset  to construct the test domain. Compared to the validation domain (MS COCO), the image resolution in YT-BB is much lower, because they are frames extracted from YouTube videos. The original YT-BB dataset contains segments extracted from 240,000 videos and approximately 5.6 million bounding box annotations for 23 categories of tracked objects. We extracted 72,372 frame crops that fall into one of our twelve categories and satisfy the size constraints. Some samples are shown in Figure 2.
Experimental Setup. Our first set of experiments aims to provide a set of baselines for the VisDA-C challenge. We perform in-domain (i.e. train and test on the same domain) experiments to obtain approximate “oracle” performance, as well as source-only (i.e. train only on the source domain) to obtain the lower bound results of no adaptation. In total, we have 152,397 images as the source domain and 55,388 images as the target domain for validation. In our in-domain experiments, we follow a 70%/30% split for training and testing, i.e., 106,679 training images, 45,718 test images for the synthetic domain and 38,772 training images, 16,616 test images for the real domain.
We first adopt the widely used AlexNet CNN architecture as the base model. The last layer is replaced with a fully connected layer with output size 12. We initialize the network with parameters learned on ImageNet , except the for the last layer, which is initialized with random weights from
. We utilize mini-batch stochastic gradient descent (SGD) and set the base learning rate to be, weight decay to be and momentum to be 0.9. We also utilize ResNext-152 ; the output dimension of the last fully connected layer is changed to 12 and initialized with “Xaiver” parameter initializer . Since the output layer is trained from scratch, we set the learning rate to be 10 times that of other layers. The learning rate is adjusted with the formula: , where will linearly change from 0 to 1 along the training process, , and . We report the accuracy of classification averaged over categories at 40k iterations.
Domain Adaptation Algorithms. We evaluate two existing domain adaptation algorithms as baselines. DAN (Deep Adaptation Network)  learns transferable features by training deep models with Maximum Mean Discrepancy  loss to align the feature distribution of source domain to target domain. In our implementation, the network architecture of DAN is extended from AlexNet , which consists of 5 convolutional layers (conv1 - conv5) and 3 fully connected layers (fc6 - fc8) and Deep CORAL (Deep Correlation Alignment)  performs deep model adaptation by matching the second-order statistics of feature distributions. The domain discrepancy is then defined as the squared Frobenius norm , where
are the covariance matrices of feature vectors from the source and target domain, respectively.
Baseline Results. Baseline results on the validation domain for classification are shown in Table 4
. “Oracle” or in-domain AlexNet performance for training and testing on the synthetic domain reaches 99.92% accuracy, and training and testing on the real validation domain leads to 87.62%. This supervised learning performance provides a loose upper bound for our adaptation algorithms. As far as unadapted source-only results on the validation dataset, AlexNet trained on the synthetic source domain and tested on the real domain obtains 28.12% accuracy, a significant drop from in-domain performance. This provides a measure of how much the domain shift affects the model. Among the tested domain adaptation algorithms, Deep CORAL improves the cross-domain performance from 28.12% to 45.53% and DAN further boosts the result to 51.62%. While their overall performance is not at the level of in-domain training, they achieve large relative improvements over the base model through unsupervised domain adaptation, improving it by 83.6% and 61.9% respectively.
In-domain oracle and source-only performance of AlexNet was similar on the test dataset to the validation dataset. Oracle performance of AlexNet is 92.08% and ResNext-152 improves the result to 93.40%. Source AlexNet achieves 30.81% mean accuracy, and DAN and Deep CORAL improve the result to 49.78% and 45.29%, respectively. As a base model, AlexNet has relatively low performance due to its simpler architecture, compared to more recent CNNs. However, the relative improvement of domain adaptation algorithms (i.e. DAN and Deep CORAL) is still large.
VisDA-C Challenge Results. We provided the labeled source/validation data to challenge participants and set up an evaluation server for the test domain. The (unlabeled) test data was only made available several weeks before the end of the challenge. Table 4 shows the top three of the many submitted domain adaptation results.
The top performing GFColourLabUEA team used the label propagation algorithm  that in turn was based on the recent
-model for semi-supervised learning and the mean teacher model . It improved their source-only ResNet-152 model from 45.3% to 92.8%, a 104% relative improvement. Their method consisted of optimizing two losses: 1) a mean cross entropy between ground truth and predictions of the so-called student network on samples from the source domain, and 2) a mean square difference between predictions of student and teacher networks on all samples from both domains. An essential step in the proposed method is to define a teacher network as having the same architecture as the student, but with its weights set to an exponential moving average of weights of the student network on previous training iterations, therefore optimizing an agreement of the student network trained in a supervised fashion with its copies from the past on samples from both domains. Moreover, the resulting loss is trained in minibatches with dropout, noise and random data augmentations to improve robustness of the resulting procedure. More specifically, the ResNet-152 network pre-trained on Imagenet was used as a base model, its last fully connected classification layer was replaced by a sequence of randomly initialized fully connected layers, whereas the first convolutional layer and the layers leading up to the first downsampling were left as is. The rest of the network is first trained in a supervised fashion on source and then adapted on target using the method described above.
The second and third-best performing teams, NLE_DA and BUPT_OVERFIT
, made use of various discrepancy measures such as maximum mean discrepancy (MMD) with different bandwidth priors measured between single resulting feature representation or combined representations from multiple layers (JMMD). Moreover, late fusion of several shallow networks built on features extracted from off the shelf deep models pretrained on Imagenet, and other recent approached to domain adaptation resulted in a substantial improvement in terms of the test score. We note that the top three teams had significantly higher source-only accuracy than our baselines, and some used model ensembling. While the top-1 team more than doubled their source performance, the two runner-ups had smaller relative improvements of 36% and 35%, respectively.
3.3 Increasing Difficulty and Future Research
The challenge participants obtained excellent performance in our test domain, on par with in-domain training. We therefore propose several ways to increase the difficulty of the task as a testbed for future research on this topic.
COCO YTBB. First, the test and validation domains can be swapped, as the MS-COCO domain turns out to be more difficult. We will shortly release ground truth labels for the test in order to enable this harder version of the task.
VisDA-C Extended. We also generate the VisDA-C-ext by adding smaller images to the real domains. In the initial version of the dataset, images whose height or width are under 70 pixels were excluded. We add these smaller or irregular-shaped images back to VisDA-C to increase its difficulty. Figure 3 shows the amount of data added in each category. In total, we add 35,591 images to the MS-COCO domain and 4,533 images to the YT-BB domain.
and ResNext-152+JMMD) on the VisDA-C training domain and test on both VisDA-C and VisDA-C-ext test domains. All the deep models are implemented in Caffe and fine-tuned from models pre-trained on the ImageNet 2012 dataset. The experimental setup for AlexNet and ResNext-152 is the same with Section 3.2. Joint Maximum Mean Discrepancy (JMMD)  is the distance between the means of kernelized representations of source and target samples, where sample representations are defined as combinations of outputs of multiple layers of a network, therefore taking cross-terms between outputs of different layers into considerations. Inspired by , we replaced the 1000-dimensional fully connected output layer with a sequence of 256-dimensional and 12-dimensional output layers (our dataset has 12 classes in total) and used combined outputs of these two layers as a feature representation.
The experimental results shown in Figure 5 indicate all three models get worse results on VisDA-C-ext, compared to VisDA-C. Moreover, the performance degradation on the validation domain (MS COCO) is larger than that of testing domain (YT-BB). The hyper-parameters of ResNext-152+JMMD are tuned on validation domain as we are not expected to use the labels in the testing domain. Figure 5 shows how the accuracy and training loss vary with iterations. The figure reveals that selecting the proper stopping criteria is important due to the drastic variation in reported accuracy. However, in the test domain, due to the lack of labels, there is no way to know how the accuracy will behave. Our conservative strategy is to choose the point where the first loss peak occurs (approximately 4,000 iterations). According to our strategy, ResNext-152+JMMD gets 64.54% and 58.37% mean accuracy on validation and test splits of VisDA-C respectfully. Hypothetically, our model can reach around 80% accuracy on the test domain of VisDA-C if we were to “cross validate on test servers” and ensemble top performing models as some teams did during the challenge.
No Imagenet Pre-training. Moreover, all participants used models trained in a supervised fashion on Imagenet as models to start from. Even though directly exploiting relations between Imagenet classes and classes used in our challenge was deliberately forbidden, the Imagenet pre-training made the resulting adaptation problem considerably easier, because resulting systems made extensive use of embedding that were designed to have high discriminative power in the first place. Considering that it is very hard to run domain adaptation synthetic-to-real challenges for domains that lack such enormous labeled datasets as Imagenet (e.g. medical imaging or decision making in robotics), it is even more important to design algorithms that can perform well without supervised pre-training on similar domains.
Variations in background/texture/POV. We also release tools, models and required metadata to enable researchers to generate their versions of the dataset to increase its difficulty or to check specific hypothesis about domain adaptation, such as robustness of these methods to degrees of variations mentioned above, or to generate datasets for different tasks such as detection.
4 VisDA-S: Semantic Segmentation
The goal of our VisDA2017 Segmentation (VisDA-S) benchmark is to test adaptation between synthetic and real dashcam footage for semantic image segmentation. The training data includes pixel-level semantic annotations for 19 classes. We also provide validation and testing data, following same protocol as for classification:
training domain (source): synthetic dashcam renderings from the GTA5 dataset along with semantic segmentation labels,
validation domain (target): a real-world collection of dashcam images from the CityScapes dataset along with semantic segmentation labels to be used for validating the unsupervised adaptation performance,
test domain (target): a different set of unlabeled, real-world images from the new Nexar dashcam dataset.
|GTA CityScapes Validation Domain|
|Source (Dilation F.E.)||30.6||21.2||44.8||10.1||4.4||15.4||12.4||1.7||75.1||13.5||58.1||38.0||0.2||67.5||9.4||5.0||0.0||0.0||0.0||21.4|
|Oracle (Dilation F.E.)||96.2||76.0||88.4||32.5||46.4||53.5||52.0||68.7||88.6||46.6||91.0||74.8||46.0||90.5||46.9||58.0||44.7||45.2||70.3||64.0|
|GTA Nexar Test Domain|
|Source (Dilation F.E.)||40.7||19.2||42.3||4.2||20.0||21.8||26.0||13.4||68.0||19.6||84.7||32.4||5.8||59.0||10.3||9.8||1.6||13.5||0.0||25.9|
The training and validation domain datasets used here are the same as those used in Hoffman et al (2016)  for their work in synthetic to real adaptation in semantic segmentation tasks.
Training Domain: Synthetic GTA5. The images in the segmentation training come from the GTA5 dataset. GTA5 consists of 24,966 high quality labeled frames from the photorealistic, open-world computer game, Grand Theft Auto V (GTA5). The frames are synthesized from a fictional city modeled off of Los Angeles, CA and are in high-resolution, 1052. All semantic segmentation labels used in the GTA5 dataset have a counterpart in the CityScapes category list for adaptation. See Figure 4 for sample training domain data.
Validation Domain: Real CityScapes. Data in the segmentation validation domain comes from the CityScapes dataset. CityScapes contains 5,000 dashcam photos separated by the individual European cities from which they were taken, with a breakdown of 2,975 training, 500 validation and 1,525 test images. Images are in high resolution, . In total, the CityScapes dataset has 34 semantic segmentation categories, of which we are interested in the 19 that overlap with the synthetic GTA5 dataset. See Figure 4 for sample validation domain data.
Test Domain: Real DashCam Images. Dashcam photos in the test domain were taken from a dataset recently released by Berkeley Deep Drive and Nexar . They were collected using the Nexar dashcam interface and manually annotated with segmentation labels. We use 1500 images of size available with annotations corresponding to the 19 categories matching GTA5 and CityScapes. Note that this data along with the annotations is part of a larger data collection effort by Berkeley Deep Drive (BDD). See Figure 4 for sample test domain data.
Domain Adaptation Algorithms. For details on the domain adaptation algorithms applied to this domain shift, we refer the reader to the original work that performed adaptation from GTA5 (synthetic) to CityScapes (real) in . The authors use the front-end dilated fully convolutional network as the baseline model. The method for domain adaptive semantic segmentation consists of both global and category specific adaptation techniques. Please see section 3 (Fully Convolutional Adaptation Models) in 
for detailed information about these techniques and their implementation. In all experiments, the Intersection over Union (IoU) evaluation metric is used to determine per-category segmentation performance.
Baseline Results. Please refer to Table 6 and Section 4.2.1 in Hoffman et al.  for full experimental results and discussion of semantic segmentation performance in GTA5CityScapes adaptation. Some relevant results are replicated here. In summary, the front-end dilation source achieves a mean IoU (mIoU) of 21.6 over all semantic categories on the val domain, compared to oracle mIoU of 64.0. The adaptation method in  improves mIoU to 25.5. A similar performance improvement is seen when adapting the GTA5 model to our challenge test domain.
Challenge Results. The winning team used a multi-stage procedure to improve performance of their adaptation pipeline. On the first stage, they trained and applied frame-level discriminator and updated pixels of target images in a way that preserves high level semantic features of objects, but makes each target frame as a whole less distinguishable from source frames. This approach resembles style-transfer initially proposed several years ago , but significantly improved since then [19, 48, 55, 23]. To perform second-stage pixel level discrimination winning team ensembled ResNet-101, ResNet-152 and SE-ResNeXt-101 with pyramid spatial pooling to obtain better feature representations and applied domain discrimination with multiple scaling parameters to get a more powerful yet stable discriminator. Other top performing teams used single-step procedures with pixel-level loss and different base models, such as DeepLab-v2, with smart balancing between weights of segmentation and adversarial losses.
In this paper, we introduce a large scale synthetic-to-real dataset for unsupervised domain adaptation, and present a detailed analysis and current state of the art performance. We also describe several modifications to the dataset that increase its difficulty as well as alternative setups that enable verification of more complex hypotheses regarding domain adaptation methods. We highly encourage researchers to work on adaptation methods that do not rely on the supervised pre-training, because there are plenty of important domains, such as medical imaging, that seriously lack labeled data, and therefore might greatly benefit from synthetic-to-real domain adaptation. Large scale synthetic-to-real datasets as the one described in this paper present an experimental setup designed for figuring out how to train these models without supervised pre-training, and therefore working on methods that perform well in practical domains that do not have large labeled datasets using simulated data, which is becoming more and more important these days. As of now the no-pre-train setup poses a substantial challenge for existing domain adaptation methods and solving it would greatly benefit the society.
Our plan for the nearest future is to opensource and support all resources, tools and dataset flavours discussed in this paper to enable the community to improve adaptation methods’ performance in diverse and not yet considered situations. We hope that this benchmark will be used by the wider research community to develop and test novel domain adaptation models using an established protocol.
-  http://data-bdd.berkeley.edu.
-  A. Bergamo and L. Torresani. Exploiting weakly-labeled web images to improve object classification: a domain adaptation approach. In Neural Information Processing Systems (NIPS), Dec. 2010.
-  F. M. Carlucci, L. Porzi, B. Caputo, E. Ricci, and S. R. Bulò. Autodial: Automatic domain alignment layers. arXiv preprint arXiv:1704.08082, 2017.
-  A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
D.-Y. Chen, X.-P. Tian, Y.-T. Shen, and M. Ouhyoung.
On visual similarity based 3d model retrieval.In Computer graphics forum, volume 22, pages 223–232. Wiley Online Library, 2003.
-  S. Choi, Q.-Y. Zhou, S. Miller, and V. Koltun. A large dataset of object scans. arXiv:1602.02481, 2016.
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson,
U. Franke, S. Roth, and B. Schiele.
The cityscapes dataset for semantic urban scene understanding.In
Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
-  G. Csurka. Domain adaptation for visual applications: A comprehensive survey. CoRR, abs/1702.05374, 2017.
-  L. Duan, D. Xu, I. Tsang, and J. Luo. Visual event recognition in videos by learning from web data. In Proc. CVPR, 2010.
-  G. French, M. Mackiewicz, and M. H. Fisher. Self-ensembling for domain adaptation. CoRR, abs/1706.05208, 2017.
-  A. Gaidon, Q. Wang, Y. Cabon, and E. Vig. Virtual worlds as proxy for multi-object tracking analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4340–4349, 2016.
-  Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409.7495, 2014.
-  L. A. Gatys, A. S. Ecker, and M. Bethge. A neural algorithm of artistic style. CoRR, abs/1508.06576, 2015.
X. Glorot and Y. Bengio.
Understanding the difficulty of training deep feedforward neural networks.In
Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249–256, 2010.
A. Graves et al.
Supervised sequence labelling with recurrent neural networks, volume 385. Springer.
-  J. Hoffman, D. Wang, F. Yu, and T. Darrell. Fcns in the wild: Pixel-level adversarial and constraint-based adaptation. arXiv preprint arXiv:1612.02649, 2016.
-  J. J. Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550–554, 1994.
-  Y. Jia. Caffe: An open source convolutional architecture for fast feature embedding. http://caffe.berkeleyvision.org/, 2013.
-  J. Johnson, A. Alahi, and F. Li. Perceptual losses for real-time style transfer and super-resolution. CoRR, abs/1603.08155, 2016.
A. Krizhevsky, I. Sutskever, and G. E. Hinton.
Imagenet classification with deep convolutional neural networks.In Advances in neural information processing systems, pages 1097–1105, 2012.
-  S. Laine and T. Aila. Temporal ensembling for semi-supervised learning. CoRR, abs/1610.02242, 2016.
-  Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
-  C. Li and M. Wand. Precomputed real-time texture synthesis with markovian generative adversarial networks. CoRR, abs/1604.04382, 2016.
-  J. J. Lim, H. Pirsiavash, and A. Torralba. Parsing IKEA Objects: Fine Pose Estimation. ICCV, 2013.
-  T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312, 2014.
-  M. Long and J. Wang. Learning transferable features with deep adaptation networks. CoRR, abs/1502.02791, 1:2, 2015.
M. Long, H. Zhu, J. Wang, and M. I. Jordan.
Deep transfer learning with joint adaptation networks.In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 2208–2217, 2017.
-  S. A. Nene, S. K. Nayar, H. Murase, et al. Columbia object image library (coil-20). 1996.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng.
Reading digits in natural images with unsupervised feature learning.
NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 5, 2011.
-  X. Peng, B. Sun, K. Ali, and K. Saenko. Learning deep object detectors from 3d models. In Proceedings of the IEEE International Conference on Computer Vision, pages 1278–1286, 2015.
-  J. Quionero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence. Dataset Shift in Machine Learning. The MIT Press, 2009.
-  E. Real, J. Shlens, S. Mazzocchi, X. Pan, and V. Vanhoucke. Youtube-boundingboxes: A large high-precision human-annotated data set for object detection in video. CoRR, abs/1702.00824, 2017.
-  S. R. Richter, V. Vineet, S. Roth, and V. Koltun. Playing for data: Ground truth from computer games. In European Conference on Computer Vision, pages 102–118. Springer, 2016.
-  G. Ros, L. Sellart, J. Materzynska, D. Vazquez, and A. Lopez. The SYNTHIA Dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2016.
-  O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
-  K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual category models to new domains. In Computer Vision–ECCV 2010, pages 213–226. Springer, 2010.
-  D. Sejdinovic, B. Sriperumbudur, A. Gretton, and K. Fukumizu. Equivalence of distance-based and rkhs-based statistics in hypothesis testing. The Annals of Statistics, pages 2263–2291, 2013.
-  T. Sim, S. Baker, and M. Bsat. The cmu pose, illumination, and expression (pie) database. In Automatic Face and Gesture Recognition, 2002. Proceedings. Fifth IEEE International Conference on, pages 53–58. IEEE, 2002.
-  H. Su, C. R. Qi, Y. Li, and L. J. Guibas. Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In Proceedings of the IEEE International Conference on Computer Vision, pages 2686–2694, 2015.
-  B. Sun, J. Feng, and K. Saenko. Return of frustratingly easy domain adaptation. arXiv preprint arXiv:1511.05547, 2015.
-  B. Sun and K. Saenko. From virtual to reality: Fast adaptation of virtual object detectors to real domains. In BMVC, volume 1, page 3, 2014.
-  B. Sun and K. Saenko. Deep CORAL: correlation alignment for deep domain adaptation. CoRR, abs/1607.01719, 2016.
-  A. Tarvainen and H. Valpola. Weight-averaged consistency targets improve semi-supervised deep learning results. CoRR, abs/1703.01780, 2017.
-  A. Tatsuma, H. Koyanagi, and M. Aono. A large-scale shape benchmark for 3d object retrieval: Toyohashi shape benchmark. In Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific, pages 1–10. IEEE, 2012.
-  T. Tommasi, T. Tuytelaars, and B. Caputo. A testbed for cross-dataset analysis. Computer Vision - ECCV 2014 Workshops. ECCV 2014, 2014.
-  E. Tzeng, C. Devin, J. Hoffman, C. Finn, X. Peng, S. Levine, K. Saenko, and T. Darrell. Towards adapting deep visuomotor representations from simulated to real environments. arXiv preprint arXiv:1511.07111, 2015.
-  E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko. Simultaneous deep transfer across domains and tasks. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
-  D. Ulyanov, V. Lebedev, A. Vedaldi, and V. S. Lempitsky. Texture networks: Feed-forward synthesis of textures and stylized images. CoRR, abs/1603.03417, 2016.
-  T. P. Vanamali, A. Godil, H. Dutagaci, T. Furuya, Z. Lian, and R. Ohbuchi. Shrec’10 track: Generic 3d warehouse. In Proceedings of the 3rd Eurographics Conference on 3D Object Retrieval, 3DOR ’10, pages 93–100, Aire-la-Ville, Switzerland, Switzerland, 2010. Eurographics Association.
-  D. Vazquez, A. M. Lopez, J. Marin, D. Ponsa, and D. Geronimo. Virtual and real world adaptation for pedestrian detection. IEEE transactions on pattern analysis and machine intelligence, 36(4):797–809, 2014.
-  Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1912–1920, 2015.
-  Y. Xiang, W. Kim, W. Chen, J. Ji, C. Choy, H. Su, R. Mottaghi, L. Guibas, and S. Savarese. Objectnet3d: A large scale database for 3d object recognition. In European Conference on Computer Vision, pages 160–176. Springer, 2016.
-  Y. Xiang, R. Mottaghi, and S. Savarese. Beyond pascal: A benchmark for 3d object detection in the wild. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2014.
-  S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. arXiv preprint arXiv:1611.05431, 2016.
-  J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593, 2017.
6 Supplementary Appendix
A Other relevant datasets
B Number of images per category in VisDA-C
Table 8 shows number of models and images in VisDA-C in detail.