It is well known that the success of machine learning methods on visual recognition tasks is highly dependent on access to large labeled datasets. Unfortunately, model performance often drops significantly on data from a new deployment domain, a problem known asdataset shift or dataset bias . Changes in the visual domain can include lighting, camera pose and background variation, as well as general changes in how the image data is collected. While this problem has been studied extensively in the domain adaptation literature , progress has been limited by the lack of large-scale challenge benchmarks. Existing cross-domain benchmarks [3, 4, 5, 6] created for evaluating domain adaptation algorithms often have limited size, low task diversity, and relatively small domain shifts.
The domain shift from simulated to real imagery can be particularly challenging. This shift occurs in many real-world situations where labeled imagery is difficult or expensive to collect but simulation is easy, such as in robotics, medical imaging, surveillance, etc. A synthetic rendering pipeline can produce virtually infinite amounts of labeled data, and can allow learning through interaction with the simulated environment [7, 8]. CAD models of a wide variety of objects are freely available in online repositories. Yet existing benchmarks for synthetic-to-real adaptation either contain a very small number of object instances , or focus on narrow tasks like eye tracking . Moreover, no existing benchmark defines multiple tasks on the same pair of synthetic and real domains.
In this paper, we introduce Syn2Real, a synthetic-to-real visual domain adaptation benchmark meant to encourage further development of robust domain transfer methods. The goal is to train a model on a synthetic “source” domain and then update it so that its performance improves on a real “target” domain, without using any target annotations. It includes three tasks, illustrated in Figure 1: the more traditional closed-set classification task with a known set of categories; the less studied open-set classification task with unknown object categories in the target domain; and the object detection task, which involves localizing instances of objects by predicting their bounding boxes and corresponding class labels.
The Syn2Real benchmark focuses on unsupervised domain adaptation (UDA) for which the target domain images are not labeled. While there may be scenarios where target labels are available (enabling supervised
domain adaptation), the purely unsupervised case is more challenging and often more realistic. For each task, we provide a synthetic source domain and two real-image target domains. Many UDA evaluation protocols do not have a validation domain and use labeled target domain data to select hyperparameters. However, assuming labeled target data goes against the UDA problem statement. For this reason, we collect twodifferent target domains, one for validation of hyperparameters and one for testing the models.
For the closed-set classification task, we generate the largest synthetic-to-real dataset to date with over 280K images in the combined training, validation and test sets. We use this dataset to hold a public challenge, inviting researchers from all over the world to compete on this task, and analyze the performance of the top submitted methods. We find that for this task, where the object categories are known ahead of time, recent UDA models that leverage CNN features pre-trained on ImageNet are able to achieve impressive adaptation results. This is surprising, considering that labels are only available on synthetic source images and the source and target domains are very different. We provide a detailed analysis and insight into these results.
We then push the boundaries of synthetic-to-real transfer beyond closed-set classification and design an open-set classification dataset, where the target domains contain images of additional unknown categories that were not present in the source dataset. We evaluate the state-of-the-art UDA methods available for this more difficult task, and find that there is still much room for improvement.
Furthermore, we propose an even more challenging object detection benchmark that covers a much more diverse set of object categories than previous syn2real detection datasets [11, 12], and show that methods that excel for adaptation of classification models completely fail when applied to recent end-to-end detection models , potentially due to very low initial performance of source models on the target data.
Our hope is that the Syn2Real benchmark will provide a useful tool for the visual domain adaptation and transfer learning community. Besides enabling large-scale evaluation for the syn-to-real domain shift, it poses a challenge for the community to develop an adaptation method that works equally well for different tasks given the same domain shift. We released our image datasets and benchmark code, and fully open sourced our 3D model collection, along with scripts and metadata required for proper rendering111http://ai.bu.edu/syn2real/.
2 Related Work
There has been a lot of prior work on visual domain adaptation, ranging from the earlier shallow feature methods [4, 14, 15] to the more recent deep adaptation approaches [16, 17].
A review of existing work in this area is beyond the scope of this paper; we refer the reader to a recent comprehensive survey .
Several most notable benchmark datasets that can be used to evaluate visual domain adaptation are summarized in Table 1. A majority of popular benchmarks lack task diversity: the most common cross-domain datasets focus on the image classification task, e.g., digits of different styles, objects  or faces  under varying conditions.
Other tasks such as open-set classification , detection , structure prediction [21, 22, 8] and sequence labeling  have been relatively overlooked.
|COIL20 ||C||1,440||20 (tools)||1||No||No|
|Office ||C||1,410||31 (office)||3||No||No|
|Caltech ||C||1,123||10 (office)||1||No||No|
|CAD-office ||C||775||20 (office)||1||Yes||No|
|Cross-Dataset ||C||70,000+||40 (mixed)||12||No||No|
|CAD-PASCAL ||D+P||12,000||20 (mixed)||2||Yes||Yes|
|NEXAR ||D||55,000||1 (car)||3||No||No|
|CVC ||D||5,616||1 (pedestrian)||2||Yes||No|
Classification Datasets. Adaptation of image classification methods has been among the most extensively studied problems of visual domain adaptation. One of the difficulties in re-using existing datasets to create multi-domain benchmarks is that the same set of categories must be shared among all domains. For digits (ten categories, 0-9), the most popular benchmark setup consists of three domains: MNIST (handwritten digits) , USPS (handwritten digits)  and SVHN (street view house numbers) . Digit images are sometimes synthetically augmented to create additional domains, for example by inverting colors or using randomly chosen backgrounds . The Office dataset  is a popular benchmark for real-world objects. It contains 31 object categories captured in three domains: office environment images taken with a high quality camera (DSLR), the same environment captured with a low quality webcam (WEBCAM), and images downloaded from the amazon.com website (AMAZON).
One problem with existing classification benchmarks is the relatively small domain shifts, such as the shift between two different sensors (DSLR vs Webcam in the Office dataset ), or between two handwritten digit datasets (MNIST vs USPS). Over time, improvements in underlying image representations and adaptation techniques have closed the domain gap on these benchmarks, and more challenging domain shifts are now needed to drive further progress.
Another issue with current datasets is their small scale. Modern computer vision methods require a lot of training data, while cross-domain datasets such as Office Dataset only contain several hundred of images. The Cross-Dataset Testbed  is a more recent classification benchmark. Its “dense” version contains 40 classes extracted from Caltech256, Bing, SUN, and Imagenet with a minimum of 20 images per class in each dataset. It is significantly larger than Office, however, some domains are fairly close as they were collected in a similar way from web search engines. On the Caltech-Imagenet shift, adaptation performance has reached close to 90% accuracy .
The popularity of the image classification task as a testbed may be due to lower effort required to engineer a good baseline model. Compared with other vision problems such as object detection, activity detection in video, or structured prediction, image classification is simpler and less computationally expensive to explore. Moreover, many state-of-the-art classification models are readily available for use as a baseline upon which adaptation can be applied. At the same time, other tasks have characteristics that present unique challenges for domain adaptation. In this work, we propose experimental setups for both the more common classification task, and the less studied open-set classification and detection tasks.
Detection Datasets. Compared to classification, cross-domain object detection is a far less studied problem. There are a few existing datasets that have been used to create multi-domain benchmarks for recognition and detection tasks in synthetic to real adaptation. Recent work in this area use adaptive learning from virtual scenarios to perform real-world pedestrian detection [11, 28]. They use scenes from the CVC virtual-world pedestrian datasets to train models that predict on the Daimer AG real-world pedestrian detection dataset . Beyond pedestrian detection, there has been some work done regarding cross-domain action detection in videos  and in facial detection (e.g. facial action units) . However, Syn2Real appears to be among the first diverse object detection datasets designed specifically for adaptive learning.
Open-set Datasets. Open-set classification was considered in the transfer learning field only very recently. As such, there is a lack of open-set datasets designed specifically for domain adaptation tasks. One recent study  generates benchmarks for open-set adaptation using the Office dataset and introduces an open-set protocol by adopting a set of 10 common classes from the Cross-Dataset Analysis Caltech dataset to use as the known sets for classification. Beyond this introduced approach, however, there do not appear to be any other datasets for the open-set adaptation task. As in the detection task, Syn2Real offers a valuable contribution to the visual domain adaptation community to study a novel problem.
Synthetic-to-Real Datasets. Synthetic data augmentation has been extensively employed in computer vision research. More specifically, 3D models have been utilized to generate synthetic images with variable object poses, textures, and backgrounds . Recent usage of 3D simulation has been extended to multiple vision tasks such as object detection [20, 32]
, pose estimation, robotic simulation , semantic segmentation . Popular 3D model databases of common objects that may be used in visual domain adaptation tasks include ObjectNet3D , ShapeNet and the related ModelNet . Table 7 in the supplementary appendix compares the Syn2Real dataset to existing synthetic object datasets.
3 Syn2Real Benchmark
We introduce the Syn2Real dataset as a large-scale testbed for studying unsupervised domain transfer and adaptation of object category models. We collect 3D models and real images of the same 12 object categories, resulting in three visual domains:
training domain (source): synthetic renderings of 3D models from different angles and with different lighting conditions,
validation domain (target): a real-image domain consisting of images cropped from the Microsoft COCO dataset ,
testing domain (target): a real-image domain consisting of images cropped from the Youtube Bounding Box dataset .
The goal is to train a vision system on images from a synthetic domain and then improve its performance on a real target domain. We propose to utilize different target domains for the validation and test splits to prevent hyper-parameter tuning on the test data. Unsupervised domain adaptation is usually done in a transductive manner, meaning that unlabeled test data is actively used to train the model. However, it is not possible to tune hyper-parameter on the test data, since it has no labels. Despite this fact, the lack of established validation sets often leads to experimental protocols where the labeled test set is used for this purpose. In our benchmark, we provide a validation set to mimic the more realistic deployment scenario where the target domain is unknown at training time and test labels are not available for hyper-parameter tuning. This setup also discourages algorithms that are designed to handle a specific target domain. The validation and test sets are different domains, so over-tuning to one can potentially degrade performance on another.
The Syn2Real benchmark consists of three tasks, each defined across the above three domains: closed-set classification (Syn2Real-C), open-set classification (Syn2Real-O), and detection (Syn2Real-D). The details of the data collection and experimental results of each of these tasks are described below.
4 Syn2Real-C: Closed-set Classification Task
Our first task is to learn to classify images from the synthetic domain, and transfer the learned models to a target real domain. For this task, we assume that the target domain images belong to the same set of known categories as the source. We collect the largest-scale dataset to date for this problem; the number of images in each category in Syn2Real-C is shown in Table2 and Figure 2 shows some examples.
4.1 Dataset Acquisition
Training Domain: CAD-Synthetic Images. We generate the source domain by rendering 3D models of the same 12 object categories as in the real data from different angles and under different lighting conditions. We obtained 1,907 3D models in total from several free online sources, the details of which are described in the supplemental material. Twenty different camera yaw and pitch combinations with four different light directions are used to render images from each model. The lighting setup consists of ambient and sun light sources in 1:3 proportion. Objects were rotated, scaled and translated to match the floor plane, duplicate faces and vertices were removed, and the camera was automatically positioned to capture the entire object with a margin around it. For textured models, we also rendered their un-textured versions with a plain grey albedo. In total, we generated 152,397 images to form the synthetic source domain.
Validation Domain: MS COCO. The validation dataset for Syn2Real is built from the Microsoft COCO dataset 
. In total, the MS COCO dataset contains 174,011 images. We used annotations provided by the COCO dataset to find and crop relevant object in each image. All images were padded by retaining an additional 50% of its cropped height and width. Padded image patches whose height or width was under 70 pixels were excluded from the dataset. In total, we collected 55,388 object images that fall into the chosen twelve categories. We took all images from each of twelve categories with the exception of the “person” category, which was limited to 4,000 images in order to balance the overall number of samples per category (the original “person” category had more than 120k images).
Testing Domain: YouTube Bounding Boxes. Due to the high number of object category labels overlapping with the other two domains, we chose the YouTube Bounding Boxes (YT-BB) dataset  to construct the test domain. Compared to the validation domain (MS COCO), the resolution of images in YT-BB is much lower, because they are frames extracted from YouTube videos. The original YT-BB dataset contains segments extracted from 240,000 videos and approximately 5.6 million bounding box annotations for 23 categories of tracked objects. We extracted 72,372 frame crops that fall into one of our twelve categories and satisfy the size constraints.
Our first set of experiments aims to provide a set of baselines for the Syn2Real-C benchmark. Below we also present and analyze the results obtained by the winners of the challenge we held on this dataset.
Experimental Setup. We perform in-domain (i.e. train and test on the same domain) experiments to obtain approximate “oracle” performance, as well as source-only (i.e. train only on the source domain) to obtain the lower bound results of no adaptation. In total, we have 152,397 images as the source domain and 55,388 images as the target domain for validation. In our in-domain experiments, we follow a 70%/30% split for training and testing, i.e., 106,679 training images, 45,718 test images for the synthetic domain and 38,772 training images, 16,616 test images for the real domain.
We first adopt the widely used AlexNet CNN architecture as the base model. The last layer is replaced with a fully connected layer with output size 12. We initialize the network with parameters learned on ImageNet , except the for the last layer, which is initialized with random weights from
. We utilize mini-batch stochastic gradient descent (SGD) and set the base learning rate to be, weight decay to be and momentum to be 0.9. We also utilize ResNext-152  as a base model; the output dimension of the last fully connected layer is changed to 12 and initialized with “Xaiver” parameter initializer . Since the output layer is trained from scratch, we set the learning rate to be 10 times that of other layers. The learning rate is adjusted with the formula: , where will linearly change from 0 to 1 along the training process, , and . The latter setup gave highest source-only results. We report the accuracy of classification at 40k iterations.
Domain Adaptation Baselines. We evaluate two existing domain adaptation algorithms as baselines. DAN (Deep Adaptation Network)  learns transferable features by training deep models with Maximum Mean Discrepancy  loss to align the feature distribution of source domain to target domain. In our implementation, the network architecture of DAN is extended from AlexNet , which consists of 5 convolutional layers (conv1 - conv5) and 3 fully connected layers (fc6 - fc8) and Deep CORAL (Deep Correlation Alignment)  performs deep model adaptation by matching the second-order statistics of feature distributions. The domain discrepancy is then defined as the squared Frobenius norm , where
are the covariance matrices of feature vectors from the source and target domain, respectively.
Baseline Results. Baseline results on the validation domain for classification are shown in Table 3
. “Oracle” or in-domain AlexNet performance for training and testing on the synthetic domain reaches 99.9% accuracy, and oracle training and testing on the real validation domain leads to 87.6%. This supervised learning performance provides a loose upper bound for our adaptation algorithms. As far as unadapted source-only results on the validation dataset, AlexNet trained on the synthetic source domain and tested on the real domain obtains 28.1% accuracy, a significant drop from in-domain performance. This provides a measure of how much the domain shift affects the model. Among the tested domain adaptation algorithms, Deep CORAL improves the cross-domain performance from 28.1% to 45.5% and DAN further boosts the result to 51.6%. While their overall performance is not at the level of in-domain training, they achieve large relative improvements over the base model through unsupervised domain adaptation, improving it by 83.6% and 61.9% respectively.
On the test domain (Table 3 bottom half), AlexNet achieved similar oracle and source-only performance to that on the validation dataset. Oracle performance of AlexNet is 92.1% and ResNext-152 improves the result slightly to 93.4%. Source AlexNet achieves 30.8% mean accuracy, and DAN and Deep CORAL improve the result to 49.8% and 45.3%, respectively. As a base model, AlexNet has relatively low performance due to its simpler architecture, compared to more recent CNNs. However, the relative improvement of domain adaptation algorithms (i.e. DAN and Deep CORAL) is still large. The source model based on ResNet-152 gives higher source accuracies then Alexnet-based and was used by all winning teams.
Challenge results. We posted the Syn2Real-C dataset and evaluation kits online and invited the public to participate in a challenge. Table 3 shows the top three of the many submitted domain adaptation results. The top performing team used the Self-Ensembling (SE in the table) algorithm . It improved their source-only ResNet-152 model from 45.3% to 91.8%, a 102% relative improvement. Authors use two networks with same architecture: student and teacher networks, and the latter has its weights set to an exponential moving average of weights
of the student network on previous training iterations. The method consisted of optimizing three losses: 1) a cross entropy between ground truth and predictions of the so-called student network on samples from the source domain, 2) a mean square difference between predictions of student and teacher networks on samples from target domain for which prediction confidence exceeds a threshold, and 3) cross entropy between uniform distribution and distribution of predicted target class labels. Therefore optimizing an agreement of the student network trained in a supervised fashion with its copies from the past on samples from target for which prediction confidence is high results in essentially bootstrapping target labels with high confidence predictions. Moreover, the resulting loss is trained in minibatches with dropout, noise and random data augmentations to improve robustness of the resulting procedure.
The second and third-best performing teams, NLE_DA and BUPT_OVERFIT
, made use of maximum mean discrepancy (MMD) with different bandwidth priors measured between single resulting feature representation or combined representations from multiple layers (JMMD). Late fusion of features extracted from off the shelf deep models pretrained on Imagenet resulted in a substantial improvement in terms of the test score.
4.3 Analysis of Results
The challenge participants obtained excellent performance in our Syn2Real-C test domain, on par with in-domain training. The self-ensembling model (SE) achieved particularly good performance. In this section, we provide a more in-depth analysis of its results. We investigate how the following factors affect its performance: number of synthetic training images, inclusion of small/irregular images in the target data, adding synthetic background clutter to synthetic images, and pre-training the CNN on ImageNet.
Effect of number of synthetic images. We performed an ablation study to understand the effective number of synthetic images required for successful adaptation. Figure 3 shows the mean per-class accuracy of the SE model for the original 152K training set and for significantly smaller sets (test and val curves). Accuracy remains stable when using only 480 source images, and starts to drop significantly at 240 images. We conclude that on the Syn2Real-C target domains, SE with Imagenet pre-training does not require a lot of synthetic data.
Effect of target image size. We generate an extended version of the target domains referred as Syn2Real-C-ext by also including target images whose height or width are under 70 pixels, to increase the difficulty of the task. In total, we add 35,591 images to the MS-COCO val domain and 4,533 images to the YT-BB test domain. Accuracy on val(ext) and test(ext) in Figure 3 drops significantly compared to the original domains, especially on val(ext) (MS-COCO). We also observe that accuracy drops off faster with smaller training set size on these more difficult target domains.
Effect of synthetic background. We also ran classification experiments on images corresponding to twelve known classes cropped from the Syn2Real-D detection dataset, as they include some background clutter, to see if that would make source images more similar to the target and therefore reduce domain shift. We observed a 3-5% drop in target performance on majority of categories and a 10-20% drop on “knife”, “skateboard” and “truck” categories. We suggest that this must be due to our synthetic background not conveying any additional information about object category, whereas in real images the background is often correlated with object class (e.g., sheep appear on green grass, etc.)
Effect of pre-training. All of the adaptation models so far have initialized the source CNN models with features pre-trained on the ImageNet dataset. As we saw above, with such strong starting features, the SE adaptation model can achieve good performance even with very few synthetic images. We also tried using random initialization (no pre-training) in several domain adaptation models but the results were below 20% accuracy. The adapted performance of the SE model not initialized with Imagenet features is reported as SE* in Table 3. Its performance highly depends on the number of synthetic images and does not saturate, however overall accuracy is very low.
5 Syn2Real-D: Detection Task
We define an object detection task on the Syn2Real. The goal is to localize each object from one of the 12 categories and predict its class and its bounding box.
Dataset Acquisition. We use 3D CAD models collected and annotated for Syn2Real-C as well as models from 33 additional categories from ShapenetCore to generate ten thousand images for the detection task, each having around ten known and ten unknown objects. Here the “known” categories are those we want to detect, whereas the “unknown” are those that appear in the background. The rendering pipeline is similar to the one described in the previous section, apart from models being scaled and then located semi-randomly across the scene. Object bounding boxes were inferred from instance segmentation maps obtained by replacing material fragment shader with a constant value shader. Occlusion rates were obtained similarly by comparing a whole scene instance segmentation map to an instance segmentation maps of the scene with all objects but one removed, an illustrating example is given in the supplementary. We used first 10K images from COCO and YTBB detection datasets as the target.
Baseline experiments. As to our knowledge there are no published baselines considering adaptation for detection of multiple object types using end-to-end detection pipelines such as SSD , Faster-RCNN  or YOLO , therefore we propose the following basic setup. We trained an SSD model consisting of a VGG16 feature extractor pretrained on Imagenet and a region proposal network predicting class labels, locations and size shifts for a total of 8k possible anchor boxes using feature maps from five last convolutional layers. We chose SSD (not YOLO or Faster-RCNN) because of its relative simplicity: if SSD does not perform well, then more compilcated models that use same building blocks will also fail. We applied multiple domain adaptation methods originally proposed for classification models to the resulting detection model as shown in Table 4. Columns of the table refer to the mean average precision of the bounding boxes that have intersection over union with ground truth bounding boxes higher then 0.5, localization and classification components of the loss, as well as mean distance between feature representations computed by corresponding adaptation methods. We computed Maximum Mean Discrepancy (MMD) between outputs of the model on source and on target as well as between the last three convolutional feature representations, whereas CORAL was computed on last convolutional feature representations considering activations for each spatial region as an individual observation when computing covariance matrices. We also used the self-ensembling method for detection by minimizing difference between output of the student model that is trained with gradient descent and a teacher model with weights equal to the exponential average of student’s weights. This difference was only considered for predictions with label confidence exceeding 0.9 threshold. We also added a class balance loss that encouraged same distribution of output classes for source and target input batches. Table 4 shows results of these experiments as well as performance of the model trained on source and right after Imagenet initialization. The student-teacher loss is equal to zero on the first iteration of SE because no predictions exceed the confidence threshold. By the end of training quite a few prediction do, and the class balance loss is also much lower.
Discussion. Even though the discrepancy measures between feature representations such as CORAL and SE losses decreased over time, target performance did not improve. We suggest that this must be due to very low (close to zero) performance of the source model on target, and therefore bootstrapping is more challenging from a poor model. This agrees with our no-pretraining results for the classification task. Moreover, feature representations given by large convolutional maps are very high-dimensional, so conventional measures such as MMD and CORAL might not be a good fit here, as these methods are usually applied to outputs of the last fully connected layers which have only a few thousand dimensions. We conclude that the detection task presents a significant future challenge for domain adaptation methods. The covariance matrices for CORAL loss were computed considering activations for each spatial region of the output as an individual observation.
|SE||0.58||0.9||2.4||0.02||3.2||5.5||(0, 0.27)||(0.04, 0.06)|
6 Syn2Real-O: Open-set Classification Task
Open-set domain adaptation considers classification when the target domain contains categories unknown (unlabelled) in the source domain. It can be viewed as an intermediate task between closed-set classification and detection.
Dataset Acquisition. We cropped object instances from the Syn2Real-D dataset described in the previous section using bounding boxes of all objects including the 33 “background” objects to build an open-set classification dataset. The 12 original categories are considered “known” and the 33 background categories are “unknown”. We only chose objects that were more than 60% non-occluded. The target domain was constructed from COCO using known images from the validation split of Syn2Real-C and 50k images from 69 other COCO categories (see Table 2 for details).
Baseline experiments. In a recent study on open-set adaptation,  suggested a shallow semi-supervised approach based on iteratively solving an assignment problem and aligning two domain instances on subsets of the Office dataset. We explored what we can obtain with more conventional deep distribution matching methods. These methods can be applied to open set domain adaptation when unknown source samples are available. The underlying idea is to match the distribution of unknown source samples with that of unknown target ones, even though the categories of unknown source and target are different. MMD  and BP  are utilized as baseline methods in . In addition, several methods for open set recognition have been proposed for the case where unknown samples are not provided during training [49, 50]. Open-set SVM 
rejects unknown samples lower than a pre-defined probability threshold, and works well when known and unknown samples belong to the same domain. Recent AODA
method utilizes adversarial approach for both domain adaptation and unknown outlier detection.
We prepared six baselines, including Source Only and distribution matching based methods. In addition, we applied self-ensembling based method (SE) 
to this setting. With regard to the six baselines, we trained models with existing distribution alignment methods as was done in the closed-set task. We present the results of Source Only, MMD and DANN based on ResNet152 that was pre-trained on ImageNet. The top fully-connected layer of the network was removed and replaced with three fully-connected layers with batch normalization
and a Leaky ReLU activation layer. We update the parameters of these newly added layers and fix the pre-trained parts of the network. In addition, we apply Open-set SVM to the features obtained by the methods. Open-set SVM does not need unknown samples for training, so we train it only on known source samples and test it on target samples, rejecting unknown samples lower than 0.1 in all experiments. To obtain the result of Oracle, we trained the networks in the same way as above.
Training Domain (CAD-synthetic) Validation Domain
Known-to-Unknown Ratio = 1:1
Known-to-Unknown Ratio = 1:10
Ablation Study. To investigate how the known-to-unknown ratio affects experimental results, we change the known-to-unknown ratio of target domain to 1:10, but keep the ratio of source domain. The experimental results listed in Table 5 show that the accuracy drops significantly after the known-to-unknown ratio is set to 1:10.
Discussion. Table 5 shows the results of the experiment. We can see the effectiveness of adaptation by comparing Source Only with MMD, DANN and SE. By applying Open-set SVM to each method, a slight improvement was observed, even though Open-set SVM does not utilize unknown samples. It explains why it has lower unknown accuracy. The accuracy was very low for some classes such as bike, person and truck. Unknown target samples seem to prevent from correctly aligning known target samples with known source samples. The mean per-category accuracy of SE is higher than other baselines, however, the overall accuracy was actually worse than others, because the SE method did not perform well at recognizing unknown objects on the target, as the table demonstrates. This also agrees with our detection experiment results, where the SE method seemed to have degraded the source model performance by not adequately handling negative boxes.
In this paper, we introduce three large scale synthetic-to-real datasets for different tasks within unsupervised domain adaptation all sharing same domain shift: object classification, open-set classification and object detection. We also present analysis of current state of the art in synthetic-to-real adaptation and explore how different properties of this setup, such as size of images, amount of training data, pre-training and background clutter influence adaptation performance.
We opensourced all resources and tools discussed in this paper to enable the community to develop and test novel domain adaptation models using an established protocol. Visit http://ai.bu.edu/syn2real for more details.
This work was supported in part by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) contract number D17PC00341222The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, or the U.S. Government, by the National Science Foundation (NSF) via awards IIS-1212928, CNS-1629700, and CCF-1723379, and by the Honda Research Institute.
-  Quionero-Candela, J., Sugiyama, M., Schwaighofer, A., Lawrence, N.D.: Dataset Shift in Machine Learning. The MIT Press (2009)
-  Csurka, G.: Domain adaptation for visual applications: A comprehensive survey. CoRR abs/1702.05374 (2017)
-  Nene, S.A., Nayar, S.K., Murase, H., et al.: Columbia object image library (coil-20). (1996)
-  Saenko, K., Kulis, B., Fritz, M., Darrell, T.: Adapting visual category models to new domains. In: Computer Vision–ECCV 2010. Springer (2010) 213–226
-  Carlucci, F.M., Porzi, L., Caputo, B., Ricci, E., Bulò, S.R.: Autodial: Automatic domain alignment layers. In: International Conference on Computer Vision. (2017)
-  Tommasi, T., Tuytelaars, T., Caputo, B.: A testbed for cross-dataset analysis. Computer Vision - ECCV 2014 Workshops. ECCV 2014 (2014)
-  Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An open urban driving simulator. In: Proceedings of the 1st Annual Conference on Robot Learning. (2017) 1–16
-  Richter, S.R., Vineet, V., Roth, S., Koltun, V.: Playing for data: Ground truth from computer games. In: European Conference on Computer Vision, Springer (2016) 102–118
Bousmalis, K., Silberman, N., Dohan, D., Erhan, D., Krishnan, D.:
Unsupervised pixel-level domain adaptation with generative
In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Volume 1. (2017) 7
-  Świrski, L., Dodgson, N.: Rendering synthetic ground truth images for eye tracker evaluation. In: Proceedings of the Symposium on Eye Tracking Research and Applications, ACM (2014) 219–222
-  Vazquez, D., Lopez, A.M., Marin, J., Ponsa, D., Geronimo, D.: Virtual and real world adaptation for pedestrian detection. IEEE transactions on pattern analysis and machine intelligence 36(4) (2014) 797–809
-  : http://data-bdd.berkeley.edu
-  Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: SSD: Single shot multibox detector. In: ECCV. (2016)
-  Bergamo, A., Torresani, L.: Exploiting weakly-labeled web images to improve object classification: a domain adaptation approach. In: Neural Information Processing Systems (NIPS). (December 2010)
-  Duan, L., Xu, D., Tsang, I., Luo, J.: Visual event recognition in videos by learning from web data. In: Proc. CVPR. (2010)
-  Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409.7495 (2014)
-  Tzeng, E., Hoffman, J., Darrell, T., Saenko, K.: Simultaneous deep transfer across domains and tasks. In: The IEEE International Conference on Computer Vision (ICCV). (December 2015)
-  Sim, T., Baker, S., Bsat, M.: The cmu pose, illumination, and expression (pie) database. In: Automatic Face and Gesture Recognition, 2002. Proceedings. Fifth IEEE International Conference on, IEEE (2002) 53–58
-  Busto, P.P., Gall, J.: Open set domain adaptation. In: The IEEE International Conference on Computer Vision (ICCV). Volume 1. (2017)
-  Peng, X., Sun, B., Ali, K., Saenko, K.: Learning deep object detectors from 3d models. In: Proceedings of the IEEE International Conference on Computer Vision. (2015) 1278–1286
-  Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.: The SYNTHIA Dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition. (2016)
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R.,
Franke, U., Roth, S., Schiele, B.:
The cityscapes dataset for semantic urban scene understanding.In: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2016)
Graves, A., et al.:
Supervised sequence labelling with recurrent neural networks. Volume 385.Springer
-  Xiang, Y., Mottaghi, R., Savarese, S.: Beyond pascal: A benchmark for 3d object detection in the wild. In: IEEE Winter Conference on Applications of Computer Vision (WACV). (2014)
-  LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11) (1998) 2278–2324
-  Hull, J.J.: A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence 16(5) (1994) 550–554
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.:
Reading digits in natural images with unsupervised feature learning.
In: NIPS workshop on deep learning and unsupervised feature learning. Volume 2011. (2011) 5
-  Vázquez, D., López, A.M., Ponsa, D.: Unsupervised domain adaptation of virtual and real worlds for pedestrian detection. In: Pattern Recognition (ICPR), 2012 21st International Conference on, IEEE (2012) 3492–3495
-  Schneider, N., Gavrila, D.M.: Pedestrian path prediction with recursive bayesian filters: A comparative study. In: German Conference on Pattern Recognition, Springer (2013) 174–183
-  Cao, L., Liu, Z., Huang, T.S.: Cross-dataset action detection. In: Computer vision and pattern recognition (CVPR), 2010 IEEE conference on, IEEE (2010) 1998–2005
-  Chu, W.S., De la Torre, F., Cohn, J.F.: Selective transfer machine for personalized facial action unit detection. In: Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, IEEE (2013) 3515–3522
-  Sun, B., Saenko, K.: From virtual to reality: Fast adaptation of virtual object detectors to real domains. In: BMVC. Volume 1. (2014) 3
-  Su, H., Qi, C.R., Li, Y., Guibas, L.J.: Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In: Proceedings of the IEEE International Conference on Computer Vision. (2015) 2686–2694
-  Tzeng, E., Devin, C., Hoffman, J., Finn, C., Peng, X., Levine, S., Saenko, K., Darrell, T.: Towards adapting deep visuomotor representations from simulated to real environments. arXiv preprint arXiv:1511.07111 (2015)
-  Xiang, Y., Kim, W., Chen, W., Ji, J., Choy, C., Su, H., Mottaghi, R., Guibas, L., Savarese, S.: Objectnet3d: A large scale database for 3d object recognition. In: European Conference on Computer Vision, Springer (2016) 160–176
-  Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., Yu, F.: ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago (2015)
-  Lin, T., Maire, M., Belongie, S.J., Bourdev, L.D., Girshick, R.B., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. CoRR abs/1405.0312 (2014)
-  Real, E., Shlens, J., Mazzocchi, S., Pan, X., Vanhoucke, V.: Youtube-boundingboxes: A large high-precision human-annotated data set for object detection in video. CoRR abs/1702.00824 (2017)
-  Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115(3) (2015) 211–252
-  Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. arXiv preprint arXiv:1611.05431 (2016)
Glorot, X., Bengio, Y.:
Understanding the difficulty of training deep feedforward neural
In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. (2010) 249–256
-  Long, M., Wang, J.: Learning transferable features with deep adaptation networks. CoRR, abs/1502.02791 1 (2015) 2
-  Sejdinovic, D., Sriperumbudur, B., Gretton, A., Fukumizu, K.: Equivalence of distance-based and rkhs-based statistics in hypothesis testing. The Annals of Statistics (2013) 2263–2291
Krizhevsky, A., Sutskever, I., Hinton, G.E.:
Imagenet classification with deep convolutional neural networks.In: Advances in neural information processing systems. (2012) 1097–1105
-  Sun, B., Feng, J., Saenko, K.: Return of frustratingly easy domain adaptation. arXiv preprint arXiv:1511.05547 (2015)
-  French, G., Mackiewicz, M., Fisher, M.H.: Self-ensembling for domain adaptation. CoRR abs/1706.05208 (2017)
-  Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems (NIPS). (2015)
-  Redmon, J., Farhadi, A.: Yolo9000: Better, faster, stronger. arXiv preprint arXiv:1612.08242 (2016)
-  Jain, L.P., Scheirer, W.J., Boult, T.E.: Multi-class open set recognition using probability of inclusion. In: European Conference on Computer Vision, Springer (2014) 393–409
-  Bendale, A., Boult, T.: Towards open world recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2015) 1893–1902
-  Saito, K., Yamamoto, S., Ushiku, Y., Harada, T.: Open set domain adaptation by backpropagation. CoRR abs/1804.10427 (2018)
-  Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning. (2015) 448–456
-  Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J.: 3d shapenets: A deep representation for volumetric shapes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2015) 1912–1920
-  Xiang, Y., Mottaghi, R., Savarese, S.: Beyond pascal: A benchmark for 3d object detection in the wild. In: IEEE Winter Conference on Applications of Computer Vision (WACV). (2014)
-  Choi, S., Zhou, Q.Y., Miller, S., Koltun, V.: A large dataset of object scans. arXiv:1602.02481 (2016)
-  Lim, J.J., Pirsiavash, H., Torralba, A.: Parsing IKEA Objects: Fine Pose Estimation. ICCV (2013)
9 Supplementary Appendix
A Other relevant datasets
In Table 1 we list the existing cross-domain datasets used for open- and closed-set object classification and object detection. We display other relevant cross-domain datasets in Table 6. Table 7 shows the comparison between Syn2Real-C and other synthetic object datasets.