Supervised learning is successful for two reasons: it is equivalent to the minimization of the training error, and stochastic gradient descent (SGD) is highly effective at minimizing training error. As a result, supervised learning is robust, reliable, and highly successful in practical applications.
Unsupervised learning is not as successful, mainly because it is not clear what the unsupervised cost function should be. The goal of unsupervised learning is often to improve the performance of a supervised learning task for which we do not have a lot of data. Due to the lack of labelled examples, unsupervised cost functions do not know which of the many possible supervised tasks we care about. As a result, it is difficult for the unsupervised cost function to improve the performance of the supervised cost function. The disappointing empirical performance of unsupervised learning supports this view.
In this paper, we present a cost function that generalizes the ideas of Casey (1986). We illustrate the idea in the setting of speech recognition. It is possible to evaluate the quality of a speech recognition system by measuring the linguistic plausibility of its typical outputs, without knowing whether these outputs are correct for their inputs. Thus, we can measure the performance of our system without the use of any input-output examples.
We formalize and generalize this idea as follows: in conventional supervised learning, we are trying to find an unknown function from to . Each training case imposes a soft constraint on :
We solve these equations by local optimization with the backpropagation algorithm(Rumelhart et al., 1986).
We now present the unsupervised cost function. Let be the true data distribution over the input-output pairs . While we do not have access to many labelled samples , we often have access to large quantities of unlabelled samples from and . This assumption is valid whenever unlabelled inputs and unlabelled outputs are abundant.
We can use uncorrelated samples from and to impose a valid constraint on :
denotes the distribution of the random variable. This constraint is valid in the following sense: if we have a function such that is true for every possible training case, then is satisfied as well. Thus, the unsupervised constraint can be seen as an additional labelled “training case” that conveys a lot of information whenever the output space is very large.
This constraint can be turned into the following cost function
We call it the output distribution matching (ODM) cost function, the cost literally matches the distributions of the outputs.
For the constraint to be highly informative of the optimal parameters, it is necessary for the output space to be large, since otherwise the ODM cost will be trivial to optimize. If the output space is large, then the ODM cost is, in principle, substantially more useful than conventional unsupervised cost functions which are unrelated to the ultimate supervised cost function. In contrast, the ODM cost is nearly guaranteed to improve the final supervised performance whenever there exists a very high performing function, because a supervised function that perfectly maps every input to its desired output is also a global minimum of the ODM cost. See Figure 1. It also means that a practitioner has a high chance of improving their supervised performance if they are able to optimize the ODM cost.
We also show how the ODM cost can be used for one-shot domain adaptation. Given a test datapoint from a distribution that is very different from the training one, our model can, in certain settings, identify a nearly unique corresponding datapoint that happens to obey the training data distribution, which can then be classified correctly. We implement this idea using a generative model as a mechanism for aligning two distributions without supervision.
This allows each new test case to be from significantly different distribution, so it is no longer necessary to assume that the test set follows a particular distribution. This capability allows our models, in principle, to work robustly with unexpected data. However, the ability to perform one-shot domain adaptation is not universal, and it can be achieved only in certain restricted settings.
2 Related work
There has been a number of publications that considered matching statistics as an unsupervised learning signal. The early work of Casey (1986) casts the problem of unsupervised OCR as a problem of decoding a cipher, where there is an unknown mapping from images to characters identities, which must be inferred from the known statistics of language. This idea has been further explored in the context of machine translation (typically in the form of matching bigrams) (Knight et al., 2006; Huang et al., 2006; Snyder et al., 2010; Ravi & Knight, 2008)
, and in unsupervised lexicon induction(Fung & McKeown, 1997; Koehn & Knight, 2002; Haghighi et al., 2008; Mikolov et al., 2013b). Notably, Knight et al. (2006) discusses the connection between unsupervised learning and language decipherment, and formulates a generative model similar to the one presented in this work.
Similar ideas have recently been proposed for domain adaptation where a model learns to map the new distribution back onto the training distribution (Tzeng et al., 2014; Gani et al., 2015). These approaches are closely related to the ODM cost, since they are concerned with transforming the new distribution back to the training distribution.
There has been a lot of other work on unsupervised learning with neural networks, which is largely concerned with the concept of “pre-training”. In pre-training, we first train the model with an unsupervised cost function, and finish training the model with the supervised cost. This concept was introduced byHinton et al. (2006) and Hinton & Salakhutdinov (2006) and later by Bengio et al. (2007). Other work used the -means objective for pre-training (Coates et al., 2011)
, and this list of references is far from exhaustive. These ideas have also been used for semi-supervised learning(Kingma et al., 2014; Rasmus et al., 2015)
and for transfer learning(Mesnil et al., 2012).
More recent examples of unsupervised pre-training are the Skip-gram model (Mikolov et al., 2013a)
and its generalization to sentences, the Skip-thought vectors model ofKiros et al. (2015). These models use well-motivated unsupervised objective functions that appear to be genuinely useful for a wide variety of language-processing tasks.
In this section, we present several approaches for optimizing the ODM cost.
3.1 ODM Costs as Generative Models
The ODM cost can be formulated as the following generative model. Let be a model fitted to the marginal , and let be the distribution:
The objective is to find a conditional (which corresponds to where are the parameters) so that matches the marginal distribution . If is an excellent model of , then the cost is precisely equivalent to the ODM cost of Eq. 3, modulo an additive constant. A similar generative model was presented by Knight et al. (2006).
It is desirable to train generative models using the variational autoencoder (VAE) ofKingma & Welling (2013). However, VAE training forces to be continuous, which is undesirable since many domains of interest are discrete. We address this by proposing the following generative model, which we term the -model:
whose cost function is
Here is continuous while and are either continuous or discrete. It is clear that this model can also be trained on labelled , whenever such data is available.
Although the negative log probability of the-model is not identical to the ODM cost, the two are closely related. Specifically, whenever the capacity of and is limited, the -model will be forced to represent the structure that is common to and in , while placing the domain specific structures into and . For example, in case of speech recognition, could contain the language model, since it is not economical to store the language model in and .
3.2 The Dual Autoencoder
We implemented the -generative model but had difficulty getting sensible results in our early experiments. We were able to get better results by designing an novel autoencoder model that is inspired by the -model, which we call the dual autoencoder, which is shown in Figure 2. It consists of two autoencoders whose “innermost weights” are shared with each other. More formally, the dual autoencoder has the form
where is the nonlinearity, and the cost is
are appropriate loss functions.
Eqs. 8 and 9 describe two autoencoders that map to and to , respectively. By sharing the weights between the autoencoders, the matrices and are encouraged to use compatible representations for the two modalities that align with in the absence of a direct supervised signal. While not principled, we found this approach to work surprisingly well on simple problems.
3.3 Generative Adversarial Networks Training
The GAN training algorithm maintains an adversary whose goal is to distinguish between samples from the data distribution and samples from the generator , and the generator learns to fool the discriminator. Eventually, if GAN training is successful, converges to a model such that the distribution of is indistinguishable from the target distribution.
The generative adversarial network offers a direct way of training generative models, and it had enjoyed considerable success in learning models of natural images (Denton et al., 2015). We use the GAN training method to train our unsupervised objective by requiring that produces samples that are indistinguishable from the target distribution.
4.1 Dual Autoencoders on MNIST Permutation Task
We begin exploring the ODM cost by selecting a simple artificial task where the distributions over and have rich internal structure but whose relationship is simple. We chose
to be the distribution of MNIST digits, andto be the distribution of MNIST digits whose pixels are permuted (with the same permutation on all digits). See Figure 2. We call it the MNIST permutation task. The goal of the task is to learn the unknown permutation using no supervised data.
We used a dual autoencoder whose architecture is 784-100-100-100-784, where the weights in the 100-100-100 subnetwork were shared between the two autoencoders. While the results were insensitive to the learning rate, it was important to use a random initialization that is substantially smaller than is typically prescribed for training neural networks models (namely, a unit Gaussian scaled by for each matrix).
We found that the dual autoencoder was easily able to recover the permutation without using any input-output examples, as shown in Figure 3.
Curiously, we found that the sigmoid nonlinearity achieved by far the best results on this task, and that the Relu and the Tanh nonlinearity were much less successful for a wide range of hyperparameters. Paradoxically, we suspect that the slower learning of the sigmoid nonlinearity was beneficial for the dual autoencoder, because it caused the “capacity” of the 100-100-100 network to grow slowly; it is plausible that the most critical learning stage took place when the 100-100-100 network had the lowest capacity.
4.1.1 Dual Autoencoder for Ciphers
We also tested the dual autoencoder on a character and a word cipher, tasks that were also considered by Knight et al. (2006). In this task, we are given two text corpora that follow the same distribution, but where the characters (or the words) of one of the corpora is scrambled. In detail, we convert a text file into a list of integers by randomly assigning characters (or words) to integers and by consistently using this assignment throughout the file. By doing this twice, we get two data sources that have identical underlying statistics but different symbols. The goal is to find the hidden correspondence between the symbols in both streams. It is not a difficult task since it can be accomplished by inspecting the frequency. Despite its simplicity, this task provides us with another way to evaluate our model.
We used the dual autoencoder architecture as before, where the input (as well as the the desired output) is represented with a bag of 10 consecutive characters from a random sentence from the text corpus. The loss function is the softmax cross entropy.
For the character-level experiments, we used the architecture 100-25-25-100, and for the word-level experiments we used 1000-100-100-1000 — where we used a vocabulary of the 1000 most frequent symbols in both data streams. Both architectures achieved perfect identification of the symbol mapping in less than 10,000 parameter updates when the sigmoid nonlinearity was used.
While the dual autoencoder was successful on these tasks, its success critically relied on the underlying statistics of the two datasets being very similar. When we trained each autoencoder on data from a different languages (namely on English and Spanish books from Project Gutenberg), it failed to learn the correct correspondence between words with similar meanings. This indicates that the dual autoencoder is sensitive to systematic differences in the two distributions.
4.2 Generative Adversarial Networks for MNIST classification
Next, we evaluated the GAN-based model on an artificial OCR task that was constructed from the MNIST dataset. In this task, we used the text from The Origin of Species downloaded from Project Gutenberg, and arbitrarily replaced each letter with a digit between 0 and 9 in a consistent manner. We call this sequence of numbers the MNIST label file. We then replaced each label with a random image of an MNIST digit of the same class, thus obtaining a very long sequence of images of MNIST digits which we call the MNIST image sequence.
The goal of this problem was to train an MLP that maps MNIST digits to 10-dimensional vectors without using any input-output examples. Instead, we train the classifier to match the 20-gram statistics of the Origin of Species, which requires no input-output examples.
We trained an MLP to map each MNIST digit into a 10-dimensional vector representing their classification. We used generative adversarial training where the adversary is a 1-d CNN to ensure sure that the distribution of 20 consecutive predictions is statistically indistinguishable from the distribution of 20 consecutive labels from the MNIST label file , where the inputs are randomly drawn from the MNIST image sequence.
A typical architecture of our classifier network was 784-300-300-10. The adversary consist of a 1D convolutional neural network with the following architecture: the first three layers are convolutional with a kernel of width 7, whit the following numbers of filter maps: 10-200-200. It is followed by global max pooling over time, and the remainder of the architecture consist of the fully connected layers 200-200-1. The nonlinearity was always ReLU, except for the output layer of both networks, which was the identity.
In these experiments, we used the squared loss and not the cross-entropy loss for the supervised objective. If we used the softmax loss, our model’s output layer would have to use the softmax layer, which we found to not work well with GAN training. The squared loss was necessary for using a linear layer for the outputs.
It is notable that GAN training was also fragile, and the sensitive hyperparameters are given here:
We initialized each matrix of the generator with a unit Gaussian scaled by ; for the discriminator we used
The is 200
Total number of parameter updates: 6000
Learning rate of generator: for 2000 updates; for another 2000 updates; and then for 2000 updates
Learning rate of discriminator:
Learning rate of the supervised squared loss:
While learning was extremely rapid, it was highly sensitive to the choice of learning rates.
While GAN training of the ODM cost alone was insufficient to find an accurate classifier the problem, we were able to achieve 4.7% test error on MNIST classification using 4 labelled MNIST examples. Here, each labelled example consists of a single labelled digit, and not a single sequence of 20 labelled digits. The result is insensitive to the specific set of 4 labelled examples. When using 2 labelled MNIST digits, the classification test error increased to 72%.
5 One-shot Learning and Domain Adaptation
If the function that we wish to learn has a small number of parameters, then it should be possible to infer it from a very small number of examples — and in some cases, from a single example.
Suppose that the training distribution is , but we are provided with a single test sample where is an unknown test distribution. If contains enough information to uniquely identify a simple function that maps samples to samples , we will be able to classify samples without any extra training: we could simply apply our existing classifier to . By inferring the function
from scratch for each test sample from scratch, the model becomes capable at correctly classifying test cases whose statistics are completely different from the training distribution, as well test cases that are as outliers that would have otherwise confused or misled the classifier. This has the potential to make our classifier significantly more robust.
We propose to solve these domain adaptation tasks using the generative model of Eq. 4. Let be a model of the data distribution and let be an unknown likelihood function. Then, given a test case from a distribution that differs from the data distribution, we can try to infer the unknown by solving the following optimization problem:
We emphasize that this optimization is run from scratch for each new test case . This approach can succeed only when the likelihood has few parameters that can be uniquely determined from the above equation using just one sample. The conditions under which this is likely to hold are discussed in sec. 6.1.
We validate these ideas on use the MNIST dataset for training and the 1-MNIST dataset for testing. The 1-MNIST dataset is obtained by replacing the intensity of each pixel with . Thus the 1-MNIST distribution differs very significantly from the MNIST distribution, and as a result, neural networks trained on MNIST are incapable of correctly classifying instances of 1-MNIST.
Our concrete model choices are the following: is implemented with a next-row-prediction LSTM with three hidden layers that has been trained to fit the MNIST distribution with the binary cross entropy loss, and is a small convolutional neural network (CNN) with one hidden layer: its first convolution has 5 filters of size 55 and its second convolution has one filter of size 55.
Given a test image from the 1-MNIST dataset, we optimize over and . We used significant L2 regularization, and optimized this cost with steps of Adagrad with 5 random restarts. The results are illustrated in Fig. 5.
While the results obtained in this section are preliminary, they show that in addition to synthesis and denoising, generative models can be used for aligning distributions and for expanding the set of distributions to which the model is applied.
6.1 When is Supervision Needed?
When does the ODM fully determine the best ? If the function class is too capable, then can map any distribution input distribution to any output distribution, which means that the ODM cost cannot possibly recover . However, since the ODM cost is consistent, it is likely to improve generalization by eliminating unsuitable functions from consideration. Further, if the output space is small, then the ODM cost will not convey enough information to be of significant help to the supervised cost.
However, there is a setting where the ODM cost function could, in principle, completely determine the best supervised function . It should succeed whenever the input distribution and the output distribution contains long-range dependencies, while is inherently incapable of modifying the long range structure by being “local”: for example, if the input is a sequence, and we change the input in one timestep, then the function’s output will also change in a small number of timestep. This setting is illustrated in Fig. 6.
ODM-based training has several limitations. The output space must be large, and the “shared hidden structure” of the two space have to be sufficiently similar. The simple techniques used in this paper are unlikely to successfully learn a mapping between two spaces if their shared structures are insufficiently similar. It is conceivable that as we develop better methods for training generative models, it will become possible to benefit from optimizing the ODM cost in a wider range of problems.
In this paper, we showed that the ODM cost provides a generic approach for unsupervised learning that is entirely consistent with the supervised cost function. Although we were not able to develop a reliable method that can train, e.g., an attention modelBahdanau et al. (2014) with the ODM objective, we presented evidence that the ODM objective provides a sensible way of training functions without the use of input-output examples. We expect better techniques for optimizing the ODM cost to make it universally applicable and useful.
- Bahdanau et al. (2014) Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
- Bengio et al. (2007) Bengio, Yoshua, Lamblin, Pascal, Popovici, Dan, Larochelle, Hugo, et al. Greedy layer-wise training of deep networks. Advances in neural information processing systems, 19:153, 2007.
- Casey (1986) Casey, Richard G. Text OCR by solving a cryptogram. International Business Machines Incorporated, Thomas J. Watson Research Center, 1986.
Coates et al. (2011)
Coates, Adam, Ng, Andrew Y, and Lee, Honglak.
An analysis of single-layer networks in unsupervised feature
International conference on artificial intelligence and statistics, pp. 215–223, 2011.
- Denton et al. (2015) Denton, Emily, Chintala, Soumith, Szlam, Arthur, and Fergus, Rob. Deep generative image models using a laplacian pyramid of adversarial networks. arXiv preprint arXiv:1506.05751, 2015.
- Fung & McKeown (1997) Fung, Pascale and McKeown, Kathleen. A technical word-and term-translation aid using noisy parallel corpora across language groups. Machine translation, 12(1-2):53–87, 1997.
- Gani et al. (2015) Gani, Yaroslav, Ustinova, Evgeniya, Ajakan, Hana, Germain, Pascal, Larochelle, Hugo, Laviolette, François, Marchand, Mario, and Lempitsky, Victor. Domain-adversarial training of neural networks. arXiv preprint arXiv:1505.07818, 2015.
- Goodfellow et al. (2014) Goodfellow, Ian, Pouget-Abadie, Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron, and Bengio, Yoshua. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672–2680, 2014.
- Haghighi et al. (2008) Haghighi, Aria, Liang, Percy, Berg-Kirkpatrick, Taylor, and Klein, Dan. Learning bilingual lexicons from monolingual corpora. In ACL, volume 2008, pp. 771–779, 2008.
- Hinton & Salakhutdinov (2006) Hinton, Geoffrey E and Salakhutdinov, Ruslan R. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507, 2006.
- Hinton et al. (2006) Hinton, Geoffrey E, Osindero, Simon, and Teh, Yee-Whye. A fast learning algorithm for deep belief nets. Neural computation, 18(7):1527–1554, 2006.
- Huang et al. (2006) Huang, Gary, Learned-Miller, Erik G, and McCallum, Andrew. Cryptogram decoding for optical character recognition. University of Massachusetts-Amherst Technical Report, 6(45), 2006.
- Kingma & Welling (2013) Kingma, Diederik P and Welling, Max. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
- Kingma et al. (2014) Kingma, Diederik P, Mohamed, Shakir, Rezende, Danilo Jimenez, and Welling, Max. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems, pp. 3581–3589, 2014.
- Kiros et al. (2015) Kiros, Ryan, Zhu, Yukun, Salakhutdinov, Ruslan, Zemel, Richard S, Torralba, Antonio, Urtasun, Raquel, and Fidler, Sanja. Skip-thought vectors. arXiv preprint arXiv:1506.06726, 2015.
- Knight et al. (2006) Knight, Kevin, Nair, Anish, Rathod, Nishit, and Yamada, Kenji. Unsupervised analysis for decipherment problems. In Proceedings of the COLING/ACL on Main conference poster sessions, pp. 499–506. Association for Computational Linguistics, 2006.
- Koehn & Knight (2002) Koehn, Philipp and Knight, Kevin. Learning a translation lexicon from monolingual corpora. In Proceedings of the ACL-02 workshop on Unsupervised lexical acquisition-Volume 9, pp. 9–16. Association for Computational Linguistics, 2002.
Mesnil et al. (2012)
Mesnil, Grégoire, Dauphin, Yann, Glorot, Xavier, Rifai, Salah, Bengio,
Yoshua, Goodfellow, Ian J, Lavoie, Erick, Muller, Xavier, Desjardins,
Guillaume, Warde-Farley, David, et al.
Unsupervised and transfer learning challenge: a deep learning approach.ICML Unsupervised and Transfer Learning, 27:97–110, 2012.
- Mikolov et al. (2013a) Mikolov, Tomas, Chen, Kai, Corrado, Greg, and Dean, Jeffrey. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013a.
- Mikolov et al. (2013b) Mikolov, Tomas, Le, Quoc V, and Sutskever, Ilya. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168, 2013b.
- Rasmus et al. (2015) Rasmus, Antti, Valpola, Harri, Honkala, Mikko, Berglund, Mathias, and Raiko, Tapani. Semi-supervised learning with ladder network. arXiv preprint arXiv:1507.02672, 2015.
Ravi & Knight (2008)
Ravi, Sujith and Knight, Kevin.
Attacking decipherment problems optimally with low-order n-gram models.In
proceedings of the conference on Empirical Methods in Natural Language Processing, pp. 812–819. Association for Computational Linguistics, 2008.
- Rumelhart et al. (1986) Rumelhart, David E., Hinton, Geoffrey E., and Williams, Ronald J. Learning representations by back-propagating errors. Nature, 323:533–536, 1986.
- Snyder et al. (2010) Snyder, Benjamin, Barzilay, Regina, and Knight, Kevin. A statistical model for lost language decipherment. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 1048–1057. Association for Computational Linguistics, 2010.
- Tzeng et al. (2014) Tzeng, Eric, Hoffman, Judy, Zhang, Ning, Saenko, Kate, and Darrell, Trevor. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.