Large and balanced datasets are normally crucial for learning classifiers. In real-world scenarios, however, one always struggles to find adequate amounts of labeled data. Even with the help of crowdsourcing, e.g., Amazon Mechanical Turk (AMT), it is often difficult to collect a large quantity of labeled instances with high quality that is necessary for training a classifier for a real-world problem. In terms of quantity, it has been shown that the amount of available training data, per object class, roughly follows a Zipf distribution . That means a small number of object classes account for most of the available training instances. In terms of quality, some domains, such as the analysis of satellite images (e.g. the comet images from Rosetta), require extensive and detailed expert user annotation [32, 48]. Large volume of LiDAR point cloud data have to be labeled before they can be used to train some classifiers . Such labeling process usually is very time consuming and requires expert-level labeling efforts or expensive equipments. Practically only a very limited portion of the data points can be obtained.
To solve the problem of lacking enough training samples, attributes [22, 30, 13] have been introduced to transfer the knowledge held by majority classes to instances in minority classes. Nevertheless, for certain tasks, such shared attributes [14, 12, 25, 10, 46] may simply be unavailable or nontrivial to define. In contrast, rather than using such a ‘learning to learn’  framework, humans can generalize and associate the similar patterns from images. This ability inspires us to circumvent the problem of lacking enough training data and solve it from a different angle: utilizing the synthetic data (e.g. the synthetic roof edges in Fig. 1) associated with real data (e.g. real roof edges in Fig. 1) in order to learn a better classifier.
The idea of associating synthetic data with real data has a long history and is associated with the development of cognitive psychology, artificial intelligence, and computer vision. For example, cognitive psychology studied a case that an infant learns to understand and imitate a facial expression from parents’ examples. In the computing domain, exemplar SVM tries to associate images with training exemplars. Different from these previous works, we create synthetic images to associate them with the real images whilst previous works associate ‘new’ real data with ‘old’ real data. By contrast, our approach is a ’free lunch’ in the sense that the proposed approach does not need any human annotation of real data, thus we could easily amplify the dataset used in training.
Learning a classifier from synthetic data is unfortunately extremely challenging due to the following reasons. Firstly, the feature distribution of synthetic data generated will shift away from that of real data. Such distribution shift is termed synthetic gap and illustrated in Fig 2. The synthetic gap is a major obstacle in using synthetic data to help learning classifiers, since synthetic data may fail to simulate the potential useful patterns of real data for training classifiers. To our knowledge, this synthetic gap problem has never been formally identified nor addressed in the literature. Secondly, since practically a small amount of labeled images may be available, it is necessary to jointly learn from synthetic and real data. The learning process must be automatically leveraged between synthetic data and real data.
To better learn a classifier from synthetic data, we propose a novel framework –Multichannel Autoencoder (MCAE) which is an extension of sparse autoencoder. The training step of MCAE is a process of bridging the synthetic gap between the real and the synthetic data by learning the mapping from (1) synthetic to real data and (2) real to real data. Critically, such mapping try to keep the real data while enforce MCAE to learn a transfer from the synthetic data to the real data. We thus can generate more synthetic data which will simulate the real data when the learned mapping is applied to them.
To facilitate the study on satellite image analysis, we introduce a new benchmark satellite roof classification (SRC) image dataset. The SRC dataset needs expert-level labeling and has unique challenges, such as satellite image blurring, building shadows, and extremely imbalanced roof class instances. To demonstrate the generality of the proposed approach, we use an additional handwritten digit dataset from the UCI machine learning repository
. In both datasets, synthetic data is generated using a parametric model of derived from real data that roughly mimics real data in terms of appearances and basic structure. Experimental results using these datasets demonstrate that better classification results can be obtained by training a classifier using the synthetic data when used by the proposed approach.
We thus highlight three contributions in this paper: (1) To the best of our knowledge, this is the first attempt to address the problem of synthetic gap, by solving which we demonstrate that the synthetic data could be used to improve the performance of classifiers. (2) We propose a Multichannel Autoencoder (MCAE) model to bridge the synthetic gap and jointly learn from both real and synthetic data. (3) Also, a novel benchmark dataset – Satellite Roof Classification (SRC) is introduced to the vision community. Such dataset is of expert-level label annotations as well as great challenges for satellite image analysis.
2 Related Work
3D image analysis. Synthetic data has been used for several 3D image analysis applications, but not for helping learn classifiers. A large number of synthetic 3D meshes in  were created by a series of mesh editing steps including subdivision, simplification, smooth, adding noise and Poisson reconstruction, in order to automatically evaluate the subjective visual quality of a 3D object. Recently, to circumvent the point labeling difficulty in a building roof classification problem using LiDAR point cloud,  explicitly indicated semantic roof points on synthetically created roof point clouds and compute point features from the synthetic point clouds. Techniques such as point cloud resampling, size normalization and mesh erosion are employed to reduce the differences between real roof and synthetic ones in data space.
Generating synthetic data. Previous method generate synthetic data in data space using tools including geometrical transformation and degradation models: In , to help off-line recognition of handwritten text, a perturbation model combined with morphological operation is applied to real data. They showed that when a moderate transformation is added to the real data, the resulting synthetic training set boost the performance. To enhance the quality of degraded document, in  degradation models such as brightness degradation, blurring degradation, noise degradation and texture-blending degradation were used to create a training dataset for a handwritten text recognition problem. The synthetic minority oversampling technique (SMOTE)  and its variants  are also powerful methods that have shown many success in various applications. However, these previous methods are relatively limited to one particular type of dataset, whilst we propose a more general methodology of generating synthetic data in this paper. We show that our methodology can be used both for SRC and handwritten digits dataset.
aims to extract the knowledge from one or more source tasks and applies the knowledge to a target task. Transfer learning has been found helpful in many real world problems, such as in sentiment classification, web page classification  and zero-shot classification of image and video data [23, 22, 16, 45, 15, 17, 33, 34, 37]. Transfer learning is categorized to three classes : inductive transfer learning, transductive transfer learning and unsupervised transfer learning. The work in this paper falls into a framework of domain adaptation [4, 44] in the transductive transfer learning. Nonetheless, different from previous domain adaptation tasks of different source and target domains, the synthetic gap is caused by the shifted feature distribution of synthetic data from real data. To solve this problem, our MCAE is developed from the idea of autoencoder.
. The hidden representation obtained by training a sparse autoencoder followed by a parameters fine tuning is useful in pre-training a deeper neural network. Recently autoencoder with its different variants[9, 18] also exhibit the success in learning and transferring sharing knowledge among data source from different domains [3, 5, 11], thus benefit other machine learning tasks.
3 Multichannel Autoencoder (MCAE)
In this section, we introduce the MCAE model as illustrated in Fig. 3. It can (1) bridge synthetic gap by minimizing the discrepancy between real and synthetic data; and (2) preserve and emphasize the potential useful patterns existed in both real and synthetic data in order to generate the better feature representations used for learning classifiers.
Essentially, synthetic and real data should have similar patterns, a natural idea of bridging synthetic gap is to learning a mapping from the synthetic data to the real data using an autoencoder, and vice versa. MCAE, hence, provides a more flexible way to learn this mapping due to the specific structure of the MCAE. There are two channels in MCAE, left one and right one. Each channel basically is an SAE, however, two channels share the same hidden layer. With this structure, MCAE basically learns two tasks in the same time. By setting different types of input and out data such as the one in denoising autoencoder, MCAE is capable for many applications. In our work, to bridge the gap between synthetic data and real data, we set the task in left channel as one that takes synthetic data as input and real data as reconstruction target, while the task in right channel use real data in both input and reconstrution target. This configuration actually is essentially meaningful that by keeping the reconstruction target identical in two channels, MCAE attempts to transform inputs in two channels towards the same target, thus minimize the discrepancy between two input dataset which are synthetic data and real data in our work.
3.1 Problem setup
Our MCAE is built on the sparse autoencoder (SAE). A basic autoencoder is a fully connected neural network with one hidden layer and can be decomposed into two parts: an encoding and a decoding process. Assume an input dataset with instances where and
is the dimension of each instance. Encoding typically transforms input data to hidden layer representation using an affine mapping squashed by a sigmoid function:
where is a sigmoid function and , is a set of unknown parameters in encoding with nodes in hidden layer.
While in decoding, with parameters , , autoencoder attempts to reconstruct the input data at the output layer by imposing another affine mapping followed by nonlinearity to hidden representation :
In above equation is viewed as a reconstruction of input . Normally, we impose . Here play a role of reconstruction target in this expression and we use notation to denote the configuration of input data short for and reconstruction target short for in an autoencoder. and indicate synthetic and real data respectively. By minimizing the reconstruction errors of all data instances, we have following objective function:
where is a weight decay term added to improve generalization of the autoencoder and leverages the importance of this term.
To avoid learning identity mapping in autoencoder, a regularization term that penalizes over-activation of the nodes in the hidden layer is added. is a sparsity parameter and is set by users and .
controls sparsity of representation in hidden layer.
Note that directly applying sparse autoencoder to our problem does not work well. For example, we can train an autoencoder purely by placing synthetic data in input layer and real data in output layer denoted as which however can not bridge the synthetic gap in our problem. Such way of reconstruction is only to complement the missing information in synthetic data from real data but not vice versa111Please refer to supplemenatry maerial for the validation.
A better representation should be reconstructed by using the information from both real and synthetic data simultaneously. Specifically, we aim at two tasks: one is which reconstructs synthetic data towards real data, and the other one is which uses identical real data for input and reconstruction target, where and indicate the left and right channel of MCAE.
3.2 MCAE model
We propose a multichannel autoencoder that uses a balance regularization to leverage the learning between two tasks, i.e. and . The structure of this new autoencoder is shown in Fig. 3. In this new structure, tasks of two channels will share the same parameters in encoding process which will enforce autoencoder to reconstruct common structure in both tasks. However, in decoding process, we divide autoencoder to two separate channels that two tasks will have their own parameters and . Dividing autoencoder to two channels at decoding layer enable a more flexible control between the two tasks. Thus autoencoder better leverage the common knowledge from the two tasks.
With two channels in the MCAE, we target to minimize the reconstruction error of two tasks together while taking into account the balance between two channels. The new objective function of the MCAE is given in the following:
is a regularization added to balance the learning rate between two channels. This regularization will have two effects on the MCAE. First, accelerates the speed of optimizing Eq. 9, since minimizing requires both and are small which in turn cause decreases faster. Second, penalizes a situation more when difference of learning error between two channels are large, so as to avoid imbalanced learning between two channels.
The minimization of Eq. 9
is achieved by back propagation and stochastic gradient descent using Quasi-Newton method. Since the regularization term is added to leverage the balance of different tasks, we have to compute the gradient of parametersand in MCAE. Please refer to the supplementary material for the detailed computation of gradients.
3.3 The advantages of MCAE over the alternative Configurations
Our MCAE enforces autoencoder to learn useful class patterns from the two tasks simultaneously. Thus it helps with capturing a common structure of synthetic and real images. Another alternative way is to concatenate the input and target of the two tasks for autoencoder. We annotate the usage of this autoencoder as Concatenate-Input Autoencoder (CIAE), since this autoencoder learns concatenated tasks at the same time. Such configurations however may result in an unbalanced optimization for these two tasks: the optimization process of one task will take over the process of the other one. It results in a biased reconstructed hidden layer of the autoencoder and thus a limited classification performance. Our experiments also validate this point in Sec. 5.
4 Generating Synthetic Data
It is an important and yet less studied topic of how to generate synthetic data. This section discusses the methodology of generating synthetic data used in our experiments. Such synthetic data have some similarities and differences with the augmented data used in deep learning e.g.. Both of synthetic data and augmented data aim at improving the generalisation capacity of classifiers. Nevertheless, the methodology of generating synthetic data brings more deformed patterns than the simply label-preserving transformations used in data augmentation.
Synthetic data are created to highlight the potential useful pattern existed in real images. We have two stages of generating synthetic data. In the first stage, for each real data used to train MCAE, a synthetic version that best matching appearance of the real data is generated; thus pairs of corresponding real and synthetic data can be used to train the MCAE. In the second stage, more synthetic data could be derived using synthetic data generated in the first stage by both interpolation and extrapolation. To distinguish the set of synthetic data used in these two stages, we use abbreviationSyn I and Syn II to represent them respectively.
In the proposed approach, the synthetic data are represented as a parametric model of a set of control points and edges associated to these points in the images. From the control points, the synthetic images could be generated to simulate the real images in terms of having the same structure or a similar appearance. Initially, the control points are selected from a synthetic prototype that generalize all images in the same class. Then the locations of the control points are iteratively optimized until convergence in order to minimize the distance between synthetic images generated by control points and the real image. We annotate the control points and edges associated to them as , where is the set of the control points, and is the set of edges connecting control points. A generalized algorithm of getting the best matching synthetic image is provided in Algorithm 1.
The synthetic prototype could be manually design or learning from given data in our work given different tasks. We will show how these two methods produce synthetic data in following two sections respectively.
4.1 Explicitly Design of the Synthetic Prototype
The generation of the synthetic prototype and control points in this scenario is inspired by the approach proposed by Zhang et al. . In their work, given enough pre-knowledge about the 3D objects, a synthetic prototype of 3D objects is explicitly designed and built. By adjusting the control points of the prototype, various kinds of 3D objects are generated. In this work, essentially, our data is very similar to theirs in a sense that roof images share a lot of characteristics such as ridge lines, valley lines and intersections between these lines, which make it possible to manually design the synthetic prototypes that characterize these patterns. Based on this observation, a synthetic roof prototype could be generated by setting the control points at the intersections of the ridge or valley lines and drawing segments connecting these control points. 222In our experiments, classification of the roof images is essentially similar to that of . Our approach recognizes the style of the roofs based on edges extracted from the roof images. For more visualisation results, please refer to our supplementary material..
In this scenario, the OptimizeControlPoints() function of Alg. 1 turns out to be a process that searches for optimal control point locations which results in a synthetic image minimizing the discrepancy between the real image and the synthetic image. A coordinate descent framework is employed to accelerate the search process. We summarize this method in Alg. 2.
4.2 Learning Synthetic Prototype from Data
In hand written digit dataset used in this work, we learn a synthetic prototype from given data. A digit prototype is generated for all images with the same digit. Congealing algorithm proposed in  is employed in this step to produce the synthetic prototypes for digits. In congealing, the project transformations are applied to images to minimize a joint entropy. Thus the prototype is considered to be an average image of all images after congealing.
Then control points are evenly sampled from the boundary detected from the prototype image. The control points needs to be mapped to each digit image in order to generate a synthetic image. To find this mapping we implement an approach that migrates the control points from the prototype images to destination image.
This point migration algorithm is based on a series of intermediate images generated in between synthetic prototype and destination image. To generate the intermediate images, we binarize all the images and the distance transformed images of the synthetic prototype and the real image are generated. Given the number of steps, an intermediate image then is generated as a binarized image of linear interpolation between two distance transformed images. In each step, the control points are snapped to the closest boundary pixels of the intermediate image. The algorithm of OptimizeControlPoints() in this situation is given in Algorithm 3, we fix the number of steps to in this algorithm.
To generate the dataset, we either interpolate or extrapolate between control points of randomly choose two synthetic images from dataset. The weights used in interpolation and extrapolation is uniformly drawn from to .
5 Experiments and Results
We validate the proposed MCAE dataset on several applications in this section. This section is organised as follows. First, in Sec 5.1, we introduce a new benchmark dataset – Satellite Roof Classification (SRC) dataset to vision community. This dataset is of high quality satellite roof class labels for the satellite images. We also briefly summarizes the handwritten digits dataset used in our paper. We explain the experimental settings in Sec. 5.2 and discuss the experimental results in Sec. 5.3.
5.1 Experiment Datasets
5.1.1 Satellite Roof Classification (SRC) Dataset
One particular interesting problem of learning classifiers from synthetic data is to analyze satellite images of the Earth. Such problems generally need very high quality (expert-level) labeled data. However, there is no previous dataset for such research purposes. To facilitate the study, a new benchmark Satellite Roof Classification (SRC) Dataset is created and used in our experiments. Given a satellite image, we employ a method described in  to crop roof images by registering artificial building footprints with the satellite image. Later, all roof images are aligned using their footprint principal directions using a method proposed in  and then are scaled to images with resolution of . Two experts are invited to contribute the labels of 6 different roof styles: flat, gable, gambrel, halfhip, hip and pyramid. Example instances of SRC dataset are shown in Fig 5.
This dataset is of great challenges for the task of visual analysis. First, qualities of the some satellite images are degraded because of significant image blurring occurred when capturing the satellite images. Second, roofs in these images are covered by various kinds of equipments such as air conditioners chimneys and water tanks, and most of roofs in our dataset are partially occluded by shadows cast by trees and some other stuffs. Such covering and shadows are great obstacles to robust visual analysis algorithms. Furthermore, the class instances of SRC dataset are naturally extreme imbalance, since some particular types of roofs (such as gambrel and pyramid) are far less than the other types in the real world. Such unbalanced distributions of data are compared in in Table I.
|Styles||Training #||Testing #||Total #|
Classification of the roof styles in the experiments are based on recognizing edges detected from the roof images. We employed the adaptive Otsu edge detection method  to extract edges from the roof images. We create synthetic prototype to characterize primary ridge lines or valley lines in a certain type of roof style. Examples of the synthetic prototypes are shown in Fig. 6. Real roof edge images and matching Syn I images are shown in Fig. 1. To create Syn II images of this dataset, synthetic images are produced by interpolation and extrapolation between images in Syn I for each roof style.
5.1.2 Handwritten Digits Dataset
We also validate our framework on handwritten digits dataset from UCI machine learning repository  which totally has instances. The handwritten digits from to in this dataset are collected from people: contributed to the training set and the other to the test set. In the experiments, the Syn I data are generated using Algorithm 3. The Syn II data of this dataset is generated using interpolation and extrapolation as described in Sec 4.
5.2 Experimental Settings
We fix the configuration of MCAE as (left channel) and (right channel). Specifically, the left channel is the reconstruction process from synthetic data to real data, while the right channel works in the same way as a standard SAE. Our experimental results will show that the representations learned in such way greatly benefit the performance of classifiers we compared.
In the experiments333All codes (including our MCAE and creating synthetic data) will be released once accepted.
, two different classifiers of utilizing learned representations from MCAE (from Sec. 3) are compared. In the first scenario, MCAE encodes input data to a representation (feature) in the hidden layer and a SVM using RBF kernel is employed in this case to show the performance of the classification. In the second scenario, MCAE takes the input images and produces the reconstructed images at the output layer. Features, in this case, are images, therefore can be fed to Convolutional Neural Network (CNN) for classification. In our experiments we build a LeNet-5 which is originally created for digit recognition. We show that using the same number of input data, the performance of the CNN prefers to the data produced by the MCAE.
We summarize all evaluations and comparisons using F-1 score, which is defined as:
5.3 Evaluation444Due to the page limit, please refer to our supplementary material for the additional experimental results.
MCAE is better than CIAE and SAE. To better evaluate the performance of the proposed MCAE, we compare MCAE with Concatenate-Input Autoencoder (CIAE)  and Sparse Autoencoder (SAE) . In these experiments, we evaluate the performance on two classifiers: a CNN using reconstructed images and SVM using encoded hidden layer representation. We present the results of these comparisons in Table II and Table III for SRC and handwritten digit datasets respectively. It could be observed from these two tables that although the performance of the CIAE is close to MCAE, the proposed MCAE gets a better performance almost in all the comparisons.
Synthetic data help learning a better classifier. We designed another group of experiments. In these experiments three different configurations of data are either reconstructed and encoded using the proposed MCAE, then used to train a CNN or a SVM in the experiments. All results from these experiments are compared in Table IV and Table V respectively. An interesting thing to notice is that in experiments, using synthetic data can only achieve the same result as using a combination of real and synthetic data. This result proves that the distribution of the real data in this case is almost overlapping with the distribution of the synthetic data.
|Feature type||Real||Syn II||Real+Syn II|
|Feature type||Real||Syn II||Real+Syn II|
MCAE bridges the synthetic gap. We compare the correlation defined as:
between real and Syn I data before and after being reconstructed by the MCAE. The intention of these comparisons is to show that real synthetic images become much more alike each other in terms of the appearance after being reconstructed by the MCAE. The results are shown in Fig. 8. It is shown that our method almost achieves correlation between real and Syn I when both data are reconstructed by the proposed MCAE. That means the proposed MCAE bridges the synthetic gap between the real data and the synthetic data. The results are shown in Fig. 7. It intuitively shows that our MCAE can help bridge the synthetic gap between real and synthetic data.
In this paper we identify the problem of synthetic gap. By solving this problem, in our experiments, we demonstrate that the synthetic data could be used to improve the performance of classifiers. To better learn classifiers from synthetic data, we have proposed a novel Multichannel autoencoder (MCAE) model. MCAE has multiple channels in its structure and is an extension from standard autoencoder. We show that MCAE not only bridges the synthetic gap between real data and synthetic data, it also jointly learns from both real and synthetic data, thus can provide more robust representation for both data. To facilitate the study on satellite image analysis, we introduce a novel benchmark dataset – SRC as one dataset used in our experiments. The proposed method has been validated on SRC and handwritten digits datasets.
-  K. Bache and M. Lichman. UCI machine learning repository, 2013.
-  G. Bal, G. Agam, O. Frieder, and G. Frieder. Interactive degraded document enhancement and ground truth generation. In Electronic Imaging 2008. International Society for Optics and Photonics, 2008.
Autoencoders, unsupervised learning, and deep architectures.Unsupervised and Transfer Learning Challenges in Machine Learning, Volume 7, page 43, 2012.
-  S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Mach. Learn., 79(1-2):151–175, May 2010.
-  Y. Bengio. Deep learning of representations for unsupervised and transfer learning. Unsupervised and Transfer Learning Challenges in Machine Learning, Volume 7, page 19, 2012.
-  J. Blitzer, M. Dredze, and F. Pereira. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Association for Computational Linguistics, 2007.
-  G. Borgefors. Distance transforms in digital images. In Computer Vision, Graphics, and Image Processing, volume 34, pages 344–371. Elsevier, 1986.
-  N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. Smote: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321–357, 2002.
-  M. Chen, Z. Xu, K. Q. Weinberger, and Fei. Marginalized denoising autoencoders for domain adaptation. In International Conference on Machine Learning, 2012.
-  A. Datta, R. Feris, and D. Vaquero. Hierarchical ranking of facial attributes. In IEEE International Conference on Automatic Face & Gesture Recognition, 2011.
-  J. Deng, Z. Zhang, E. Marchi, and B. Schuller. Sparse autoencoder-based feature transfer learning for speech emotion recognition. In Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on, pages 511–516. IEEE, 2013.
S. Dhar, V. Ordonez, and T. L. Berg.
High level describable attributes for predicting aesthetics and
IEEE Conference on Computer Vision and Pattern Recognition, 2011.
-  A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth. Describing objects by their attributes. In IEEE Conference on Computer Vision and Pattern Recognition, 2009.
-  Y. Fu, T. Hospedales, T. Xiang, and S. Gong. Attribute learning for understanding unstructured social activity. In European Conference on Computer Vision, 2012.
-  Y. Fu, T. M. Hospedales, T. Xiang, Z. Fu, and S. Gong. Transductive multi-view embedding for zero-shot recognition and annotation. In European Conference on Computer Vision, 2014.
-  Y. Fu, T. M. Hospedales, T. Xiang, and S. Gong. Learning multi-modal latent attributes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013.
-  Y. Fu, Y. Yang, T. Hospedales, T. Xiang, and S. Gong. Transductive multi-label zero-shot learning. In British Machine Vision Conference, 2014.
-  X. Glorot, A. Bordes, and Y. Bengio. Domain adaptation for large-scale sentiment classification: A deep learning approach. In International Conference on Machine Learning, 2011.
-  H. Han, W.-Y. Wang, and B.-H. Mao. Borderline-smote: a new over-sampling method in imbalanced data sets learning. In Advances in intelligent computing, pages 878–887. Springer, 2005.
-  H. He, Y. Bai, E. A. Garcia, and S. Li. Adasyn: Adaptive synthetic sampling approach for imbalanced learning. In Neural Networks, 2008. IJCNN 2008.(IEEE World Congress on Computational Intelligence). IEEE International Joint Conference on, pages 1322–1328. IEEE, 2008.
-  A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Neural Information Processing Systems, 2012.
-  C. H. Lampert, H. Nickisch, and S. Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In IEEE Conference on Computer Vision and Pattern Recognition, 2009.
-  C. H. Lampert, H. Nickisch, and S. Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013.
-  Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 1998.
-  J. Liu, B. Kuipers, and S. Savarese. Recognizing human actions by attributes. In IEEE Conference on Computer Vision and Pattern Recognition, 2011.
-  T. Malisiewicz, A. Gupta, and A. A. Efros. Ensemble of exemplar-svms for object detection and beyond. In IEEE International Conference on Computer Vision, 2011.
-  E. G. Miller, N. E. Matsakis, and P. A. Viola. Learning from one example through shared densities on transforms. In Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on, volume 1, pages 464–471. IEEE, 2000.
-  J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng. Multimodal deep learning. In International Conference on Machine Learning, 2011.
-  N. OTSU. A threshold selection method from gray level histograms. IEEE Trans. Syst. Man Cybern., 9(1):62–66, 1979.
-  M. Palatucci, G. Hinton, D. Pomerleau, and T. M. Mitchell. Zero-shot learning with semantic output codes. In Neural Information Processing Systems, 2009.
S. J. Pan and Q. Yang.
A survey on transfer learning.
IEEE Transactions on Data and Knowledge Engineering, 22(10):1345–1359, 2010.
-  J. A. Richards. Remote Sensing Digital Image Analysis – An introduction. Springer Berlin Heidelberg, 2013.
-  M. Rohrbach, M. Stark, and B. Schiele. Evaluating knowledge transfer and zero-shot learning in a large-scale setting. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
-  M. Rohrbach, M. Stark, G. Szarvas, I. Gurevych, and B. Schiele. What helps where – and why? semantic relatedness for knowledge transfer. In IEEE Conference on Computer Vision and Pattern Recognition, pages 910–917, 2010.
-  R. Salakhutdinov, A. Torralba, and J. Tenenbaum. Learning to share visual appearance for multiclass object detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2011.
-  K. Sarinnapakorn and M. Kubat. Combining subclassifiers in text categorization: A dst-based solution and a case study. IEEE Trans. on Knowl. and Data Eng., 19(12):1638–1651, Dec. 2007.
-  R. Socher, M. Ganjoo, H. Sridhar, O. Bastani, C. D. Manning, and A. Y. Ng. Zero-shot learning through cross-modal transfer. In Neural Information Processing Systems, 2013.
-  S. Thrun. Learning To Learn: Introduction. Kluwer Academic Publishers, 1996.
L. van der Maaten and G. Hinton.
Visualizing high-dimensional data using t-SNE.Journal of Machine Learning Research, 2008.
-  T. Varga and H. Bunke. Effects of training set expansion in handwriting recognition using synthetic data. In In 11th Conf. of the International Graphonomics Society. Citeseer, 2003.
-  T. Varga and H. Bunke. Comparing natural and synthetic training data for off-line cursive handwriting recognition. In Frontiers in Handwriting Recognition, 2004. IWFHR-9 2004. Ninth International Workshop on, pages 221–225. IEEE, 2004.
-  P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning, 2011.
-  P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. The Journal of Machine Learning Research, 11:3371–3408, 2010.
-  K. Weinberger, A. Dasgupta, J. Langford, A. Smola, and J. Attenberg. Feature hashing for large scale multitask learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, pages 1113–1120, New York, NY, USA, 2009. ACM.
-  B. Yao, X. Jiang, A. Khosla, A. L. Lin, L. J. Guibas, and L. Fei-Fei. Action recognition by learning bases of action attributes and parts. In IEEE International Conference on Computer Vision, 2011.
-  F. X. Yu, L. Cao, R. S. Feris, J. R. Smith, and S.-F. Chang. Designing category-level attributes for discriminative visual recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2013.
-  X. Zhang and G. Agam. A learning-based approach for automated quality assessment of computer-rendered images. In IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, 2012.
-  X. Zhang, G. Agam, and X. Chen. Alignment of 3d building models with satellite images using extended chamfer matching. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2014.
-  X. Zhang, A. Zang, G. Agam, and X. Chen. Learning from synthetic models for roof style classification in point clouds. In Proceedings of the 22nd ACM SIGSPATIAL international conference on Advances in geographic information systems. ACM, 2014.
-  Q.-Y. Zhou and U. Neumann. Fast and extensible building modeling from airborne lidar data. In Proceedings of the 16th ACM SIGSPATIAL international conference on Advances in geographic information systems, page 7. ACM, 2008.
6.1 Optimization of MCAE
With two branches in the MCAE, we target to minimize the reconstruction error of two tasks together while taking into account the balance between two branches. The new objective function of the YMAE is given in the following:
is a regularization added to balance the learning rate between two branches. This regularization will have two effects on the YMAE. First, accelerates the speed of optimizing Eq. 9, since minimizing requires both and are small which in turn cause decrease faster. Second, penalize a situation more when difference of learning error between two branches are large, so as to avoid imbalanced learning between two branches.
The minimization of Eq. 9 is achieved by back propagation and stochastic gradient descent using Quasi-Newton method. In the MCAE, with balance regularization added to the objective, the only difference as opposed to sparse autoencoder is the gradient computation of unknown parameters and . We clarify these differences in the following equations:
The exact form of gradients of and varies according to different sparsity regularization used in the framework.
7 Generating synthetic data
An example in Fig. 9 shows how control points are moved from source image (the one with blue boundary) to destination image (the one with red boundary). It could be observed that most of the control points are moved from the source image to corresponding locations on destination image. In this step, it is not necessary for all control points accurately move to exact corresponding location on the destination image. Our goal is just to use these migrated control points to generate synthetic data which will roughly mimic the real data. Our MCAE later will rectify the difference between synthetic data and real data.
7.1 Further Validation for MCAE
Note that directly applying sparse autoencoder to our problem does not work well. For example, we can train an autoencoder purely by placing synthetic data in input layer and real data in output layer which however can not bridge the synthetic gap in our problem. Such way of reconstruction is only to complement the missing information in synthetic data from real data. On the contrary, reconstructed real data using such SAE will add unnecessary information and noisy patterns to reconstructed data.
To validate this point, we extend the experiments of two datasets and show that SAE can not bridge the gap of synthetic gap. The results are shown in Fig. 10. The reconstructed data of SAE have lower average divergence than the other methods. That means, SAE performances worse than MCAE in bridging the synthetic gap.