Semi-Supervised Learning with the Deep Rendering Mixture Model

12/06/2016 ∙ by Tan Nguyen, et al. ∙ Baylor College of Medicine Rice University 0

Semi-supervised learning algorithms reduce the high cost of acquiring labeled training data by using both labeled and unlabeled data during learning. Deep Convolutional Networks (DCNs) have achieved great success in supervised tasks and as such have been widely employed in the semi-supervised learning. In this paper we leverage the recently developed Deep Rendering Mixture Model (DRMM), a probabilistic generative model that models latent nuisance variation, and whose inference algorithm yields DCNs. We develop an EM algorithm for the DRMM to learn from both labeled and unlabeled data. Guided by the theory of the DRMM, we introduce a novel non-negativity constraint and a variational inference term. We report state-of-the-art performance on MNIST and SVHN and competitive results on CIFAR10. We also probe deeper into how a DRMM trained in a semi-supervised setting represents latent nuisance variation using synthetically rendered images. Taken together, our work provides a unified framework for supervised, unsupervised, and semi-supervised learning.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 7

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Humans are able to learn from both labeled and unlabeled data. Young infants can acquire knowledge about the world and distinguish objects of different classes with only a few provided “labels”. Mathematically, this poverty of input implies that the data distribution contains information useful for inferring the category posterior

. The ability to extract this useful hidden knowledge from the data in order to leverage both labeled and unlabeled examples for inference and learning, i.e. semi-supervised learning, has been a long-sought after objective in computer vision, machine learning and computational neuroscience.

In the last few years, Deep Convolutional Networks (DCNs) have emerged as powerful supervised learning models that achieve near-human or super-human performance in various visual inference tasks, such as object recognition and image segmentation. However, DCNs are still far behind humans in semi-supervised learning tasks, in which only a few labels are available. The main difficulty in semi-supervised learning in DCNs is that, until recently, there has not been a mathematical framework for deep learning architectures. As a result, it is not clear how DCNs encode the data distribution, making combining supervised and unsupervised learning challenging.

Recently, the Deep Rendering Mixture Model (DRMM) [13, 14] has been developed as a probabilistic graphical model underlying DCNs. The DRMM is a hierarchical generative model in which the image is rendered via multiple levels of abstraction. It has been shown that the bottom-up inference in the DRMM corresponds to the feedforward propagation in the DCNs. The DRMM enables us to perform semi-supervised learning with DCNs. Some preliminary results for semi-supervised learning with the DRMM are provided in [14]. Those results are promising, but more work is needed to evaluate the algorithms across many tasks.

In this paper, we systematically develop a semi-supervised learning algorithm for the Non-negative DRMM (NN-DRMM), a DRMM in which the intermediate rendered templates are non-negative. Our algorithm contains a bottom-up inference pass to infer the nuisance variables in the data and a top-down pass that performs reconstruction. We also employ variational inference and the non-negative nature of the NN-DRMM to derive two new penalty terms for the training objective function. An overview of our algorithm is given in Figure 1. We validate our methods by showing state-of-the-art semi-supervised learning results on MNIST and SVHN, as well as comparable results to other state-of-the-art methods on CIFAR10. Finally, we analyze the trained model using a synthetically rendered dataset, which mimics CIFAR10 but has ground-truth labels for nuisance variables, including the orientation and location of the object in the image. We show how the trained NN-DRMM encodes nusiance variations across its layers and show a comparison against traditional DCNs.

2 Related Work

We focus our review on semi-supervised methods that employ neural network structures and divide them into different types.

Autoencoder-based Architectures:

Many early works in semi-supevised learning for neural networks are built upon autoencoders

[1]

. In autoencoders, the images are first projected onto a low-dimensional manifold via an encoding neural network and then reconstructed using a decoding neural network. The model is learned by minimizing the reconstruction error. This method is able to learn from unlabeled data and can be combined with traditional neural networks to perform semi-supervised learning. In this line of work are the Contractive Autoencoder

[16]

, the Manifold Tangent Classifier, the Pseudo-label Denoising Auto Encoder

[9], the Winner-Take-All Autoencoders [11], and the Stacked What-Where Autoencoder [20]. These architectures perform well when there are enough labels in the dataset but fail when the number of labels is reduced since the data distribution is not taken into account. Recently, the Ladder Network [15] was developed to overcome this shortcoming. The Ladder Network approximates a deep factor analyzer where each layer in the model is a factor analyzer. Deep neural networks are then used to do approximate bottom-up and top-down inference.

Deep Generative Models:

Another line of work in semi-supervised learning is to use neural networks to estimate the parameters of a probabilistic graphical model. This approach is applied when the inference in the graphical model is hard to derive or when the exact inference is computationally intractable. The Deep Generative Model family is in this line of work

[7, 10].

Both Ladder Networks and Deep Generative Models yield good semi-supervised learning performance on benchmarks. They are complementary to our semi-supervised learning on DRMM. However, our method is different from these approaches in that the DRMM is the graphical model underlying DCNs, and we theoretically derive our semi-supervised learning algorithm as a proper probabilistic inference against this graphical model.

Generative Adversarial Networks (GANs): In the last few years a variety of GANs have achieved promising results in semi-supervised learning on different benchmarks, including MNIST, SVHN and CIFAR10. These models also generate good-looking images of natural objects. In GANs, two neural networks play a minimax game. One generates images, and the other classifies images. The objective function is the game’s Nash equilibrium, which is different from standard object functions in probabilistic modeling. It would be both exciting and promising to extend the DRMM objective to a minimax game as in GANs, but we leave this for future work.

3 Deep Rendering Mixture Model

The Deep Rendering Mixture Model (DRMM) is a recently developed probabilistic generative model whose bottom-up inference, under a non-negativity assumption, is equivalent to the feedforward propagation in a DCN [13, 14]. It has been shown that the inference process in the DRMM is efficient due to the hierarchical structure of the model. In particular, the latent variations in the data are captured across multiple levels in the DRMM. This factorized structure results in an exponential reduction of the free parameters in the model and enables efficient learning and inference. The DRMM can potentially be used for semi-supervised learning tasks [13].

Definition 1 (Deep Rendering Mixture Model).

The Deep Rendering Mixture Model (DRMM)

is a deep Gaussian Mixture Model (GMM) with special constraints on the latent variables. Generation in the DRMM takes the form:

(1)
(2)
(3)

where is the layer, is the object category, are the latent (nuisance) variables at layer , and are parameter dictionaries that contain templates at layer . Here the image is generated by adding isotropic Gaussian noise to a multiscale “rendered” template .

In the DRMM, the rendering path is defined as the sequence from the root (overall class) down to the individual pixels at . The variable is the template used to render the image, and represents the sequence of local nuisance transformations that partially render finer-scale details as we move from abstract to concrete. Note that the factorized structure of the DRMM results in an exponential reduction in the number of free parameters. This enables efficient inference, learning, and better generalization.

A useful variant of the DRMM is the Non-Negative Deep Rendering Mixture Model (NN-DRMM), where the intermediate rendered templates are constrained to be non-negative. The NN-DRMM model can be written as

(4)

It has been proven that the inference in the NN-DRMM via a dynamic programming algorithm leads to the feedforward propagation in a DCN. This paper develops a semi-supevised learning algorithm for the NN-DRMM. For brevity, throughout the rest of the paper we will drop the NN.

Sum-Over-Paths Formulation of the DRMM: The DRMM can be can be reformulated by expanding out the matrix multiplications in the generation process into scalar products. Then each pixel intensity is the sum over all active paths leading to that pixel of the product of weights along that path. The sparsity of controls the number fraction of active paths. Figure 2 depicts the sum-over-paths formulation graphically.

Figure 2: (A)The Sum-over-Paths Formulation of the DRMM. Each rendering path contributes only if it is active (green)[14]. While exponentially many possible rendering paths exist, only a very small fraction are active. (B) Rendering from layer in the DRMM. (C) Inference in the Nonnegative DRMM leads to processing identical to the DCN.

4 DRMM-based Semi-Supervised Learning

4.1 Learning Algorithm

Our semi-supervised learning algorithm for the DRMM is analogous to the hard Expectation-Maximization (EM) algorithm for GMMs

[2, 13, 14]. In the E-step, we perform a bottom-up inference to estimate the most likely joint configuration of the latent variables and the object category given the input image. This bottom-up pass is then followed by a top-down inference which uses to reconstruct the image and compute the reconstruction error . It is known that when applying a hard EM algorithm on GMMs, the reconstruction error averaged over the dataset is proportional to the expected complete-data log-likelihood. For labeled images, we also compute the cross-entropy

between the predicted object classes and the given labels as in regular supervised learning tasks. In order to further improve the performance, we introduce a Kullback-Leibler divergence penalty

on the predicted object class and a non-negativity penalty on the intermediate rendered templates at each layer into the training cost objective function. The motivation and derivation for these two terms are discussed in section 4.2 and 4.3 below. The final objective function for semi-supervised learning in the DRMM is given by , where , , and are the weights for the cross-entropy loss , reconstruction loss , variational inference loss , and the non-negativity penalty loss , respectively. The losses are defined as follows:

(5)
(6)
(7)
(8)

Here, is an approximation of the true posterior . In the context of the DRMM and the DCN, is the SoftMax activations, is the class prior, is the set of labeled images, and if and otherwise. The

operator is applied element-wise and equivalent to the ReLu activation function used in DCNs.

During training, instead of a closed-form M step as in EM algorithm for GMMs, we use gradient-based optimization methods such as stochastic gradient descent to optimize the objective function.

4.2 Variational Inference

The DRMM can compute the most likely latent configuration given the image , and therefore, allows the exact inference of . Using variational inference, we would like the approximate posterior to be close to the true posterior by minimizing the KL divergence . It has been shown in [3] that this optimization is equivalent to the following:

(9)

where . A similar idea has been employed in variational autoencoders [6]

, but here instead of using a Gaussian distribution,

is a categorical random variable. An extension of the optimization 

9 is given by:

(10)

As has been shown in [4], for this optimization, there exists a value for such that latent variations in the data are optimally disentangled.

The KL divergence in Eqns. 9 and 10 results in the loss in the semi-supervised learning objective function for DRMM (see Eqn. 5). Similarly, the expected reconstruction error corresponds to the loss in the objective function. Note that this expected reconstruction error can be exactly computed for the DRMM since there are only a finite number of configurations for the class

. When the number of object classes is large, such as in ImageNet

[17] where there are 1000 classes, sampling techniques can be used to approximate . From our experiments (see Section 5), we notice that for semi-supervised learning tasks on MNIST, SVHN, and CIFAR10, using the most likely predicted in the bottom-up inference to compute the reconstruction error yields the best classification accuracies.

4.3 Non-Negativity Constraint Optimization

In order to derive the DCNs from the DRMM, the intermediate rendered templates must be non-negative[14]. This is necessary in order to apply the max-product algorithm, wherein we can push the max to the right to get an efficient message passing algorithm. We enforce this condition in the Top-Down inference of the DRMM by introducing new non-negativity constrains into the optimization 9 and 10. There are various well-developed methods to solve optimization problems with non-negativity constraints. We employ a simple but useful approach, which adds an extra non-negativity penalty, in this case, , into the objective function. This yields an unconstrained optimization which can be solved by gradient-based methods such as stochastic gradient descent. We cross-validate the penalty weight .

5 Experiments

We evaluate our methods on the MNIST, SVHN, and CIFAR10 datasets. In all experiments, we perform semi-supervised learning using the DRMM with the training objective including the cross-entropy cost, the reconstruction cost, the KL-distance, and the non-negativity penalty discussed in Section 4.1. We train the model on all provided training examples with different numbers of labels and report state-of-the-art test errors on MNIST and SVHN. The results on CIFAR10 are comparable to state-of-the-art methods. In order to focus on and better understand the impact of the KL and NN penalties on the semi-supervised learning in the DRMM, we don’t use any other regularization techniques such as DropOut or noise injection in our experiments. We also only use a simple stochastic gradient descent optimization with exponentially-decayed learning rates to train the model. Applying regularization and using better optimization methods like ADAM [6] may help improve the semi-supervised learning performance of the DRMM. More model and training details are provided in the Appendix.

5.1 Mnist

MNIST dataset contains 60,000 training images and 10,000 test images of handwritten digits from 0 to 9. Each image is of size 28-by-28. For evaluating semi-supervised learning, we randomly choose images with labels from the training set such that the amounts of labeled training images from each class are balanced. The remaining training images are provided without labels. We use a 5-layer DRMM with the feedforward configuration similar to the Conv Small network in [15]

. We apply batch normalization on the net inputs and use stochastic gradient descent with exponentially-decayed learning rate to train the model.

Table 1 shows the test error for each experiment. The KL and NN penalties help improve the semi-supervised learning performance across all setups. In particular, the KL penalty alone reduces the test error from to when . Using both the KL and NN penalties, the test error is reduced to , and the DRMM achieves state-of-the-art results in all experiments. 111The results for improved GAN is on permutation invariant MNIST task while the DRMM performance is on the regular MNIST task. Since the DRMM contains local latent variables and at each level, it is not suitable for tasks such as permutation invariant MNIST We also analyze the value that the KL and NN penalties add to the learning. Table 2 reports the reductions in test errors for when using the KL penalty only, the NN penalty only, and both penalties during training. Individually, the KL penalty leads to significant improvements in test errors ( reduction in test error when ), likely since it helps disentangle latent variations in the data. In fact, for a model with continuous latent variables, it has been experimentally shown that there exists an optimal value for the KL penalty such that all of the latent variations in the data are almost optimally disentangled [4]. More results are provided in the Appendix.

Model Test error (%) for a given number of labeled examples
DGN [7] -
catGAN [19] - -
Virtual Adversarial [12] - -
Skip Deep Generative Model [10] - -
LadderNetwork [15] -
Auxiliary Deep Generative Model [10] - -
ImprovedGAN [18] -
DRMM 5-layer
DRMM 5-layer + NN penalty
DRMM 5-layer + KL penalty
DRMM 5-layer + KL and NN penalties
Table 1: Test error for semi-supervised learning on MNIST using unlabeled images and labeled images.
Model Test error reduction (%)
DRMM 5-layer + NN penalty
DRMM 5-layer + KL penalty
DRMM 5-layer + KL and NN penalties
Table 2: The reduction in test error for semi-supervised learning on MNIST when using the KL penalty, the NN penalty, and both of them. The trainings are for unlabeled images and labeled images

5.2 Svhn

Like MNIST, the SVHN dataset is used for validating semi-supervised learning methods. SVHN contains 73,257 color images of street-view house number digits. For training, we use a 9-layer DRMM with the feedforward propagation similar to the Conv Large network in [15]. Other training details are the same as for MNIST. We train our model on and show state-of-the-art results in Table 3.

Model Test error (%) for a given number of labeled examples
500 1000 2000
DGN [7]
Virtual Adversarial [12]
Auxiliary Deep Generative Model [10]
Skip Deep Generative Model [10]
ImprovedGAN [18]
DRMM 9-layer + KL penalty
DRMM 9-layer + KL and NN penalty
Table 3: Test error for semi-supervised learning on SVHN using unlabeled images and labeled images.

5.3 Cifar10

We use CIFAR10 to test the semi-supervised learning performance of the DRMM on natural images. For CIFAR10 training, we use the same 9-layer DRMM as for SVHN. Stochastic gradient descent with exponentially-decayed learning rate is still used to train the model. Table 4 presents the semi-supervised learning results for the 9-layer DRMM for images. Even though we only use a simple SGD algorithm to train our model, the DRMM achieves comparable results to state-of-the-art methods ( versus test error when as with the Ladder Networks). For semi-supervised learning tasks on CIFAR10, the Improved GAN has the best classification error ( and test errors when ). However, unlike the Ladder Networks and the DRMM, GAN-based architectures have an entirely different objective function, approximating the Nash equilibrium of a two-layer minimax game, and therefore, are not directly comparable to our model.

Model Test error (%)
4000 8000
Ladder network [15] -
CatGAN [19] -
ImprovedGAN [18]
DRMM 9-layer + KL penalty
DRMM 9-layer + KL and NN penalty
Table 4: Test error for semi-supervised learning on CIFAR10 using unlabeled images and labeled images.

5.4 Analyzing the DRMM using Synthetic Imagery

In order to better understand what the DRMM learns during training and how latent variations are encoded in the DRMM, we train DRMMs on our synthetic dataset which has labels for important latent variations in the data and analyze the trained model using linear decoding analysis. We show that the DRMM disentangles latent variations over multiple layers and compare the results with traditional DCNs.

Dataset and Training:

The DRMM captures latent variations in the data [13, 14]. Given that the DRMM yields very good semi-supervised learning performance on classification tasks, we would like to gain more insight into how a trained DRMM stores knowledge of latent variations in the data. To do such analysis requires the labels for the latent variations in the data. However, popular benchmarks such as MNIST, SVHN, and CIFAR10 do not include that information. In order to overcome this difficulty, we have developed a Python API for Blender, an open-source computer graphics rendering software, that allows us to not only generate images but also to have access to the values of the latent variables used to generate the image.

The dataset we generate for the linear decoding analysis in Section 5.4 mimics CIFAR10. The dataset contains 60K gray-scale images in 10 classes of natural objects. Each image is of size 32-by-32, and the classes are the same as in CIFAR10. For each image, we also have labels for the slant, tilt, x-location, y-location and depth of the object in the image. Sample images from the dataset are given in Figure 3.

Figure 3: Samples from the synthetic dataset used in linear decoding analysis on DRMM and DCNs

For the training, we split the dataset into the training and test set, each contains 50K and 10K images, respectively. We perform semi-supervised learning with labeled images and images without labels using a 9-layer DRMM with the same configuration as in the experiments with CIFAR10. We train the equivalent DCN on the same dataset in a supervised setup using the same number of labeled images. The test errors are reported in Table 5.

Model Test error (%)
50K
Conv Large 9-layer
DRMM 9-layer + KL and NN penalty
Table 5: Test error for training on the synthetic dataset using unlabeled images and labeled images.

Linear Decoding Analysis:

We applied a linear decoding analysis on the DRMMs and the DCNs trained on the synthetic dataset using . Particularly, for a given image, we map its activations at each layer to the latents variables by first quantizing the values of latent variables into 10 bins and then classifying the activations into each bin using first ten principle components of the activations. We show the classification errors in Figure 4.

Figure 4: Linear decoding analysis using different numbers of labeled data . The horizontal dashed line represents random chance.

Like the DCNs, the DRMMs disentangle latent variations in the data. However, the DRMMs keeps the information about the latent variations across most of the layers in the model and only drop those information when making decision on the class labels. This behavior of the DRMMs is because during semi-supervised learning, in addition to object classification tasks, the DRMMs also need to minimize the reconstruction error, and the knowledge of the latent variation in the input images is needed for this second task.

6 Conclusions

In this paper, we proposed a new approach for semi-supervised learning with DCNs. Our algorithm builds upon the DRMM, a recently developed probabilistic generative model underlying DCNs. We employed the EM algorithm to develop the bottom-up and top-down inference in DRMM. We also apply variational inference and utilize the non-negativity constraint in the DRMM to derive two new penalty terms, the KL and NN penalties, for the training objective function. Our method achieves state-of-the-art results in semi-supervised learning tasks on MNIST and SVHN and yields comparable results to state-of-the-art methods on CIFAR10. We analyzed the trained DRMM using our synthetic dataset and showed how latent variations were disentangled across layers in the DRMM. Taken together, our semi-supervised learning algorithm for the DRMM is promising for wide range of applications in which labels are hard to obtain, as well as for future research.

References

Appendix A Model Architectures and Training Details

Model Optimiser/Hyper-parameters Dataset DRMM architecture
DRMM SGD [8] MNIST Input 784 (flattened 28x28x1).
of ConvSmall learning rate 0.2 - 0.0001 E-step Bottom-Up Conv 32x5x5 (Full),
[15] Maxpool 2x2,

over 500 epochs

Conv 64x3x3 (Valid), 64x3x3 (Full),
Maxpool 2x2,
, , Conv 128x3x3 (Valid), 10x1x1 (Valid),
Meanpool 6x6,
, Softmax.
batch size BatchNorm after each Conv layer. ReLU activation.
Classes 10.
E-step Top-Down DRMM Top-Down Reconstruction. No BatchNorm.
Upsampling nearest-neighbor.
DRMM SGD [8] SVHN Input 3072 (flattened 32x32x3).
of ConvLarge learning rate 0.2 - 0.0001 CIFAR10 E-step Bottom-Up Conv 96x3x3 (Half), 96x3x3 (Full), 96x3x3 (Full),
[15] Maxpool 2x2,
over 500 epochs Conv 192x3x3 (Valid), 192x3x3 (Full), 192x3x3 (Valid),
Maxpool 2x2,
, , Conv 192x3x3 (Valid), 192x1x1 (Valid), 10x1x1 (Valid),
Meanpool 6x6,
, Softmax.
batch size BatchNorm after each Conv layer. ReLU activation.
Classes 10.
E-step Top-Down DRMM Top-Down Reconstruction. No BatchNorm.
Upsampling nearest-neighbor.
Table 6: Details of the model architectures and trainings in the paper.

The details of model architectures and trainings in this paper are provided in Table 6. The models are trained using Stochastic Gradient Descent [8]

with exponentially-decayed learning rate. All convolutions are of stride one, and poolings are non-overlapping. Full, half, and valid convolutions follow the standards in Theano. Full convolution increases the image size, half convolution reserves the image size, and valid convolution decreases the image size. The mean and variance in batch normalizations

[5] are kept track during the training using exponential moving average and used in testing and validation. The implementation of the DRMM generation process can be found in Section B. The set of labeled images is replicated until its size is the same as the size of the unlabeled set (60K for MNIST, 73,257 for SVHN, and 50K for CIFAR10). In each training iteration, the same amounts of the labeled and unlabeled images (half of the batch size) are sent into the DRMM. The batch size used is 100. The values of hyper-parameters provided in Table 6 are for in case of MNIST, in case of SVHN, and in case of CIFAR10. is the number of labeled images used in training. Also, DRMM of ConvSmall is the DRMM whose E-step Bottom-Up is similar to the ConvSmall network [15], and DRMM of ConvLarge is the DRMM whose E-step Bottom-Up is similar to the ConvLarge network [15]. Note that we only apply batch normalization after the convolutions, but not after the pooling layers.

Appendix B Generation Process in the Deep Rendering Mixture Model

As mentioned in Section 3 of the paper, generation in the DRMM takes the form:

where is the layer, is the object category, are the latent (nuisance) variables at layer , and are parameter dictionaries that contain templates at layer . The image is generated by adding isotropic Gaussian noise to a multiscale “rendered” template . When applying the hard Expectation-Maximization (EM) algorithm, we take the zero-noise limit. Here, where

is a vector of binary switching variables that select the templates to render and

is the vector of rendering positions. Note that (see Figure 2B) and where UL, UR, LL and LR stand for upper left, upper right, lower left and lower right positions, respectively. As defined in [14], the intermediate rendered image is given by:

(11)
(12)
(13)
(14)

where is a masking matrix, is the set of core templates of size

(without any zero-padding and translation) at layer

, is a set of zero-padding operators, and is a set of translation operators to position . Elements of and are indexed by . Also, are the same for in the same channel of the intermediate rendered image . Note that in the main paper, we call and intermediate rendered templates.

The DRMM layer can be implemented using convolutions of filters , or equivalently, deconvolutions of filters . and are used to select rendering templates and positions to render, respectively. In the E-step Top-Down Reconstruction, and estimated in the E-step Bottom-Up are used instead.

References