Deep Trans-layer Unsupervised Networks for Representation Learning

09/27/2015 ∙ by Wentao Zhu, et al. ∙ Institute of Computing Technology, Chinese Academy of Sciences 0

Learning features from massive unlabelled data is a vast prevalent topic for high-level tasks in many machine learning applications. The recent great improvements on benchmark data sets achieved by increasingly complex unsupervised learning methods and deep learning models with lots of parameters usually requires many tedious tricks and much expertise to tune. However, filters learned by these complex architectures are quite similar to standard hand-crafted features visually. In this paper, unsupervised learning methods, such as PCA or auto-encoder, are employed as the building block to learn filter banks at each layer. The lower layer responses are transferred to the last layer (trans-layer) to form a more complete representation retaining more information. In addition, some beneficial methods such as local contrast normalization and whitening are added to the proposed deep trans-layer networks to further boost performance. The trans-layer representations are followed by block histograms with binary encoder schema to learn translation and rotation invariant representations, which are utilized to do high-level tasks such as recognition and classification. Compared to traditional deep learning methods, the implemented feature learning method has much less parameters and is validated in several typical experiments, such as digit recognition on MNIST and MNIST variations, object recognition on Caltech 101 dataset and face verification on LFW dataset. The deep trans-layer unsupervised learning achieves 99.45 class and 75.98 87.10

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 5

page 11

page 22

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Almost all high-layer tasks such as classification, recognition and verification require us to design fine representations for their specific aims. For classification of images taken from the wild, numerous factors in the environment, such as different lighting conditions, occlusions, corruptions and deformations, lead to large amount of intra-class variability in images. Good representations should reduce such non-informative intra-class variability, whilst preserving discriminative information across classes. However, designing good feature representations is a quite tough and difficult procedure for pattern recognition tasks, which is a hot topic in machine learning field.

The research of feature representations mainly contains two aspects, hand-crafted feature designing and automatic feature learning. Researchers and engineers made enormous efforts to devise robust feature representations at their own domains a decade ago. Many successful features are proposed such as SIFT Lowe_IJCV04 and HoG Triggs_CVPR05

features in computer vision domain. However, these hand-crafted features have poor transfer ability over domains. Novel features need to be redesigned elaborately when the domain of application is changed. The other way is representation learning, which is a quite prevalent topic after deep learning coming out

Hinton_SCI06 . Nevertheless, these fully learned representations by multi-layer unsupervised learning followed by a fine-tuning procedure have too many parameters to be tuned, and require much expertise knowledge and sophisticated hardware support to train a long time.

In this paper, we demonstrate a novel trans-layer neural network with quite simple and the most classical unsupervised learning method, PCA or auto-encoder, as the building block. Different from the PCANet Mayi_PCANet , a one-by-one two layer PCA network, the responses of the previous layer of our model are concatenated to that of the last layer to form a more complete representation. Such trans-layer connections make up the rapid information loss in the cascade unsupervised learning effectively. In addition, the local contrast normalization LeCun_ICCV09 and whitening are added in our trans-layer unsupervised network to boost its learning ability, which are commonly used in deep neural networks Coates_KMEANSJ . The difference between the implemented deep trans-layer unsupervised network and conventional networks is that the deep trans-layer unsupervised network requires no back propagation information to fine-tune the feature banks.

Experimental results indicate that the implemented trans-layer connection scheme boosts the deep trans-layer unsupervised network effectively, and commonly used local contrast normalization and whitening also contribute to the performances. The demonstrated deep trans-layer unsupervised network is validated on digit recognition and object recognition tasks. Quite surprisingly, the stacked conventional unsupervised learning with trans-layer representations achieves 99.45 % accuracy on MNIST dataset, and 67.11 % accuracy on 15 samples per class and 75.98 % accuracy on 30 samples per class on Caltech 101 dataset LiFeifei_CVPR04 .

We will start by reviewing the related works on feature learning and representation in Section 2. Then the idea of the deep trans-layer unsupervised network, including the pre-processing and trans-layer unsupervised learning, is illustrated detailedly in Section 3. How to use the deep trans-layer unsupervised network to extract features and tackle applications is also described in Section 3. The experimental results and comparative analysis on MNIST, MNIST variations and Caltech 101 datasets are presented on Section 4. Finally, discussions, conclusions and the future work are summarized in Section 5 and Section 6.

2 Related works

Much research has been conducted to pursuit a good representation by manually designing elaborative low-level features, such as LBPH feature LBPH , SIFT feature Lowe_IJCV04 and HoG feature Triggs_CVPR05 in computer vision field. However, these hand-crafted features cannot be easily adapted to new conditions and tasks, and redesigning them usually requires novel expertise knowledge and tough studies.

Learning good feature representations is probably a promising way to handle the required elaborative and expertise problem in devising hand-crafted features. Much recent work in machine learning has focused on how to learn good feature representations from massive unlabelled data, and great progresses have been made by the methods

Hinton_NIPS12 ; LeCun_ICML13

. The main idea of deep models is to learn multi-level features at different layers. High-level features generated in the upper-layer are expected to extract more complex and abstract semantics of data, and more invariance to intra-class variability, which is quite useful to high-level tasks. These deep learning methods typically learn multi-level features by greedily “pre-training” each layer using the specific unsupervised learning, and then fine-tuning the pre-trained features by stochastic gradient descent (SGD) method with supervised information

Hinton_SCI06 , LeCun_ICCV09 . However, these deep architectures have numerous parameters such as the number of features to learn and parameters of unsupervised learning in each layer. Besides, the SGD also has various parameters such as momentum, weight decay rate, learning rate, and extra parameters including the Dropout rate or DropConnet rate in recently proposed convolution deep neural networks (ConvNets) Hinton_NIPS12 ; LeCun_ICML13 .

There is also some work on conventional unsupervised learning methods with only single layer Coates_KMEANSJ ; Andrew_NIPS11 . The main idea of these methods is to learn over complete representations with dense features. Although these methods have made much progress on benchmark datasets with almost no hyper parameters, these single layer unsupervised representational learning models require over complete features of dimensions as high as possible, and the parameters need to be elaborately chosen in order to obtain satisfactory results Coates_KMEANSJ .

A major drawback of deep learning methods with fine-tuning for stacking representations is the consuming of expensive computational resources and high complexities of the models. One intuition is that, since the elaborately learned features are quite similar to some conventional unsupervised features, such as wavelets, PCA and auto encoder, why not jump over the tough and time-consuming parameter fine-tuning procedure and take those features stacked as the representation directly. Furthermore, more robust invariant features can be better devised other than various pooling strategies. Wavelet scattering networks (ScatNet) are such networks with pre-fixed wavelet filter banks in the deep convolution architectures Mallat_PAMI13 . The ScatNets have quite solid mathematical analysis of their rotation and translation invariants at any scale. More surprisingly, superior performance over ConvNet and DNNs is obtained by the ScatNets with no fine-tuning phase. However, the ScatNet is shown to have inferior performance in large intra-class variability including great illumination changing and corruption such as face related tasks Mayi_PCANet .

The other non-propagation deep network with pre-fixed feature banks is the PCANet Mayi_PCANet . The PCANet uses two layer cascaded linear networks with naïve PCA filter banks to extract more complex features. The output of the two layer cascaded PCA network is processed by the quantized histogram units. The PCANet presents a superior or highly comparable performance over other methods such as ScatNet Mallat_PAMI13 , ConvNet Hinton_NIPS12 and HSC Yukai_CVPR11

, especially in face recognition tasks with large occlusion, illumination, expression and pose changes. In addition, the PCANet has quite fewer parameters, much faster learning speed and much more reliable than the currently widely researched ConvNets and DNNs, which is much more convenient and more practical to applications

Mayi_PCANet . However, the cascaded PCA structure in PCANet will face great information loss and corruption after multi-layer transformation, which will be illustrated in Section 5. The current prevalent deep networks are also probably facing the same problem that lower layer’s discriminative information will be lost after layers’ transformation. This leads to inferior results of PCANet on conventional object recognition tasks.

This paper will tackle the multi-layer information loss problem by demonstrating a novel trans-layer structure based on multi-layer conventional unsupervised filter banks. The local contrast normalization and whitening operations are applied to ameliorate the unsupervised learning in the deep trans-layer network. Thus the trans-layer unsupervised network forms a more complete and effective representation, whilst retaining the advantages such as fewer parameters, faster learning speed and more reliable performance. Also, histogram operation is adopted to preserve translation and rotation invariance after binary quantization. Experimental results confirm that the deep trans-layer unsupervised network boosts the performance of conventional unsupervised learning, and it learns effective feature representations that achieve 99.45 % accuracy on MNIST dataset, and 67.11 % accuracy on 15 samples per class and 75.98 % accuracy on 30 samples per class on Caltech 101 dataset.

3 Deep trans-layer unsupervised network

In this section, we present a novel framework, the deep trans-layer unsupervised network, for feature learning and representation. The framework of proposed deep trans-layer unsupervised network is illustrated in Figure 1. The procedures of the deep trans-layer unsupervised network is similar to other commonly used frameworks in computer vision LeCun_CVPR10 and feature learning work Andrew_ICML09 as well. Different from the traditional methods, the deep trans-layer unsupervised network utilizes the unsupervised learning methods, such as the PCA or auto encoder, to learn the local receptive filter banks, and needs no fine-tuning procedure to adjust those local filter banks. Besides, the previous layer’s unsupervised feature maps are concatenated to the last layer to form a much more completed representation, which is shown quite effective for the following tasks.

Figure 1: The framework of the deep trans-layer unsupervised network, which is a three-layer neural network, including the first unsupervised learning layer, the second unsupervised learning layer and the trans-layer.

As a high level, the proposed deep trans-layer unsupervised performs the following steps to learn a feature representation.

The first layer

  • Extract random patches from the training images.

  • Apply local contrast normalization (LCN) and whitening operations to each extracted patches.

  • Learn the first layer’s filter banks by the unsupervised learning, such as PCA or auto encoder, on the extracted patches.

  • Use the learned feature banks to generate the feature maps of each training image.

The second layer

  • Extract random patches from the obtained feature maps in the first layer.

  • Apply local contrast normalization and whitening operations to each extracted patches from feature maps.

  • Learn the second layer’s filter banks by the unsupervised learning on the extracted feature map patches.

Given test images, we can extract stacked and concatenated feature representations and apply them to classification.

The first layer

  • Apply the unsupervised learning feature banks the first layer learned to each test image to generate its feature maps after local contrast normalization and whitening processing operations.

The second layer

  • Apply the unsupervised learning feature banks the second layer learned to each first layer feature maps after local contrast normalization and whitening processing operations.

The output layer

  • Generate the response maps by amalgamating the first layer feature maps with the second layer feature maps.

  • Use binary coding operation to facilitate the next histogram procedure.

  • Compose block-wise histogram of the coded response maps of each test image as its feature representation.

  • Apply dimensionality reduction methods or distance metrics to the learned deep trans-layer unsupervised feature representations, or directly train a classifier based the representations to tackle the applications.

In the following paragraphs, we will describe the components of the above network pipeline in more details.

3.1 Representation learning

The structure of deep trans-layer unsupervised network is partially similar to convolution neural network where convolution operations are done in small patches. In deep trans-layer unsupervised network, the local filters are learned by unsupervised learning, such as PCA or auto encoder, which requires no fine tuning process through error back propagation.

In each unsupervised layer (the first and second layer), the system begins with extracting a large number of random patches from unlabelled input images. Suppose the images used here are all gray images. Each patch has a receptive field size or dimension of -by-. If the images are color images with channels, the patch dimension is -by--by-. Just process the other channels the same as following procedures step by step independently. Then a dataset of patches is constructed, , where stands for the th patch extracted from the input images in the first layer or the feature maps in the second layer. Sequentially, we apply the preprocessing of local contrast normalization (LCN) and whitening, and then unsupervised learning in the first and second layer, respectively.

3.1.1 Local contrast normalization and whitening

In the pre-processing of each layer’s unsupervised learning, we perform several simple operations effectively in the implemented network.

The first is local contrast normalization (LCN) LeCun_ICCV09 . For each local patch in the extracted patch dataset X, we normalize the patch

by subtracting its mean and dividing by its standard deviation as,

(1)

where is a constant integer to make the model more robust, which is commonly manipulated in practice. In our work, is set to 10.

The LCN has explicit explanations both in physics and physiology. The mean of local patch stands for local brightness, and the standard deviation represents contrast normalization. By LCN, the illumination and material optimal property effects are removed. On the other hand, the LCN has an effect similar to lateral inhibition found in real neurons. The LCN operator inhabits the responses within each local patch, whilst activating responses in the same location of these patches.

Following the LCN, whitening is the second preprocessing method for each unsupervised learning layer. Whitening is commonly used in various application and is a decorrelation operator, which reduces redundant representation of images. The whitening operator transforms the patches as,

(2)

where Y is formed by patches , stands for covariance function and the size of output data is ,

is the eigenvalue decomposition function,

D and U

are eigenvalues and eigenvectors respectively,

is set as here to make the operator more robust. The ZCA whitening also has biological explanation and has been proved its effectiveness by a lot of work.

3.1.2 Trans-layer unsupervised learning

After pre-processing for each layer, unsupervised learning, such as PCA or auto encoder, is used to learn feature banks in the trans-layer network.

The first unsupervised layer

Assuming that the number of feature banks in the first layer is , flatten each pre-processed patch

extracted from input images, and put the flattened vectors together. Extracted patch matrix from the first layer will be obtained as

(3)

If the used unsupervised learning is PCA, the PCA unsupervised learning aims to

(4)

where is the PCA transformation weights, and

is an identity matrix of size

. We get the solution of the above PCA constraints as

(5)

where are the first eigenvectors of with

largest variances. Then, we should make these eigenvectors changing back to

dimensions to facilitate the next PCA feature mapping operation.

The first PCA unsupervised layer’s feature maps are generated by applying these filters to convolute with the input images. For each input image, feature maps can be obtained by

(6)

where

stands for an input image with zero padded to make

have the same size as the input image, and stands for a convolution operator.

If the used unsupervised learning is auto encoder, the auto encoder unsupervised learning aims to

(7)

where is the auto encoder transformation weights, is the encoder bias,

is the tradeoff between errors and model complexity, the used activation function

is the hyperbolic tangent function, is the decoder bias, is a column vector of size full of elements , and is obtained by randomly turning of the elements in into . Then the encoder weights and bias are calculated by stochastic gradient decent method. The solution of the above de-noising auto encoders is as

(8)

where is the weights of the th neuron in the encoder layer, and stands for the bias of the th neurons in the encoder layer. Then, we should make these weights changing back to dimensions to facilitate the next feature mapping operation.

The first de-noising auto encoders unsupervised layer’s feature maps are generated by applying the encoder layer to convolute with the input images. For each input image, feature maps can be obtained by

(9)

where stands for an input image with zero padded to make have the same size as the input image, and is a matrix of the same size as full of elements .

The second unsupervised layer

Patches should be extracted from feature maps obtained from the first unsupervised layer. These patches are also the pre-processed by LCN and whitening operations. By flattening these patches, the patch matrix is constructed as

(10)

where stands for flattened patch from the first layer’s feature maps after pre-processing, and m stands for the number of extracted patches, and , stand for the size of extracted patches.

The unsupervised learning method is applied to the patch matrix the same as the first layer. Assuming that the number of the second layer’s filters is , its solution can be obtained as 5 or 8 by solving 4 or 7. The second layer’s feature maps based on the first layer’s feature map are calculated as 6 or 9.

If the used unsupervised learning is PCA, the second layer’s unsupervised feature maps are calculated by

(11)

where is the zero padded image of the first layer’s feature map, and is the second layer’s feature bank weights.

If the unsupervised learning is auto encoder, the second layer’s unsupervised feature maps are calculated by

(12)

where and are the weights and bias of the th neuron in the second auto encoder layer.

For an input image, we get feature maps after the two convolution layers by concatenating the first layer maps to the second layer. That is

(13)

where stands for the th feature map of the first layer, and stands for the th feature map of the second layer for the th feature map of the first layer .

3.1.3 Block-wise histogram

The third layer of the network is illustrated in Figure 2. The third layer is quite similar to that of LBPH LBPH . For an input training image, the first step is to encoder the real valued feature maps with binary values, 0 and 1. The operation converts these feature maps into binary images.

Figure 2: Illustration showing the output layer of deep trans-layer unsupervised network. We first encoder the output of the trans-layer unsupervised learning. Then compress each binarized feature maps into one feature map with pixels in an integer range from to . Thus, compressed feature maps will be obtained for each image. Then use a

receptive field and stride

to generate blocks for each compressed feature map. For each block, histogram with bins is calculated. Finally, concatenate the histograms to form the feature representation of deep trans-layer unsupervised network.

The second step is to compress these binary feature maps by quantizing each binary feature maps. Here the number of second layer filters is set as 8, and the number of first layer filters is also set as 8. That is, we compress each binary feature maps into one feature map, and the compressed feature maps have pixel values from 0 to 255. Then we get compressed feature maps for each training image as

(14)

where represents the th compressed feature map.

The third step is to construct block-wise histogram illustrated as the third procedure in Figure 2. First, we should partition each compressed feature map. Assuming that the size of compressed feature is , and the size of block is with strides , each compressed feature map is partitioned into blocks. For all the compressed feature maps of each input image, we get blocks as

(15)

where stands for the th block. Next step is to build histograms in each of the blocks. In the deep trans-layer PCA network, we set the number of bins in the histogram to . It means that each integer of the pixel values is set as a bin and a sparse vector representing the histogram is constructed. Then concatenate these histograms to form a more complete representation of the input image as

(16)

where stands for histogram operators. Then we use the deep trans-layer unsupervised network representation of each training image to learn a dimensional reduction weight, or to train a classifier to tackle the next applications directly.

3.2 Feature extraction and classification

After the above representation learning phase, the unsupervised convolution feature banks have been learned. Given a test image, the feature extraction is to map the image to trans-layer representation with

dimensions by these feature banks.

3.2.1 Convolutional extraction and block-wise histogram

Given a test image, we should zero-pad the image for the sake of keeping the same size between feature maps and input image. Then the pre-processing of LCN and whitening is applied to the zero-padded image. Feature maps are calculated by the first unsupervised layer with filters of size .

For the second unsupervised layer, the procedure is the same as the first layer. First, zero-pad each feature map. Then, pre-process these feature maps. Next, put these feature maps into unsupervised learning of feature banks of size . Finally, merge the first layer’s feature maps with the second layer’s feature maps to form a more complete trans-layer representation.

The last phase is block-wise histogram. Binary encoding is used to tackle these real valued feature maps. And then form a compact representation by quantizing each binary feature maps into one feature map with pixel values from 0 to . Then partition these compact feature maps into blocks, and construct histogram representation within each block. Concatenate these histograms to form the deep trans-layer unsupervised representation of translation and rotation invariance.

3.2.2 Classification

For real applications in the next experiments, classifiers or dimension reduction methods are following the deep trans-layer unsupervised representation. Due to the relative high dimensions of this representation, whitening PCA (WPCA) is used to reduce the representation in object recognition tasks. The WPCA is conducted by conventional PCA weights weighted by the inverse of their corresponding squared root energies. Also, the deep trans-layer unsupervised representation can be directly used to train a classifier to tackle recognition tasks. In our experiments of digit recognition and object recognition, a simple linear SVM classifier with no parameter tuned is used following the deep trans-layer unsupervised network. The parameter, the cost factor , in the used linear SVM software kit LIBLINEAR is 1 as default LibLinear .

4 Experimental results

In the experiment, we will validate the performance of deep trans-layer PCA network (using PCA unsupervised learning the local receptive features) and deep trans-layer auto encoder network (using de-noising auto encoder unsupervised learning the local receptive features). The proposed deep trans-layer unsupervised network has two key phases, LCN in pre-processing and trans-layer concatenation. We will validate the two phases in the MNIST variations data set Bengio_ICML07 using the deep tran-layer PCA network. Also, other parameters such as block size and stride size are chosen through cross validation or validation set. Benchmark experiments are conducted on digit recognition of MNIST LeCun_98 and MNIST variations Bengio_ICML07 data sets, and object recognition of Caltech 101 data set LiFeifei_CVPR04 .

4.1 Effect of local contrast normalization

We first validate the effect of LCN followed by conventional PCA in our network. Experiments about deep trans-layer PCA network with LCN and without LCN are conducted on the MNIST variations data sets.

The MNIST data set contains 60,000 training samples and 10,000 test samples of gray images with size pixels. The data set is a subset of NIST, which contains hand-written digits in real world. The recognition targets have been size-normalized and centered in the images LeCun_98 . MNIST variations data sets are created by applying simple controllable factor variations on MNIST digits Bengio_ICML07 . The data sets are effective ways to study the invariance ability of representation learning methods. Details about recognition tasks, numbers of classes and samples are included in Table 1.

Data sets Recognition tasks #Classes #Train-Test
MNIST Handwritten digits from 0 to 9 10 60000-10000
mnist-basic Smaller subset of MNIST 10 12000-50000
mnist-rot Same as basic with rotation 10 12000-50000
mnist-back-rand Same as basic with random background 10 12000-50000
mnist-back-image Same as basic with image background 10 12000-50000
mnist-rot-back-image Same as basic with rotation and image background 10 12000-50000
rectangles Tall or wide rectangles 2 1200-50000
rectangles-image Same as rect. with image background 2 12000-50000
ConvexNonConvex Convex or non-convex shapes 2 8000-50000
Table 1: Details of the 9 used data sets for digit recognition on MNIST LeCun_98 and MNIST variations Bengio_ICML07

The parameters of the block size, the stride size and the patch size are determined on validation set experimentally. The validation sets are typically partitioned from the training set in consistence with the related work Bengio_ICML07 . On the digit recognition tasks, the filter size is set to pixels, the number of filters is set to ==8 and the size of strides is the half size of the block. For MNIST, MNIST basic, mnist-rotation, and rectangles-image data sets, the block size is pixels. For mnist-back-rand, mnist-back-image and mnist-rot-back-image data sets, the block size is pixels. For rectangles data set, the block size is pixels. For convex data set, the block size is pixels. These validated tiny parameters are fixed in the following digit recognition tasks. A simple linear SVM with default parameters is connected to the deep trans-layer PCA representation to do recognition task LibLinear .

Two groups of experiments are conducted on MNIST basic, mnist-rotation, mnist-back-rand, mnist-back-image-rotation data sets to validate the effect of LCN. The performance of deep trans-layer PCA Network with and without LCN is reported in Table 2.

Methods Basic Rot. Bk-rand Bk-im Bk-im-rot
Without LCN 1.03 6.40 5.09 10.51 34.79
With LCN 0.98 6.36 5.22 9.95 33.55
Table 2: Comparison of classification error rates (%) on MNIST variations data sets using deep trans-layer PCA network with LCN and without LCN.

From Table 2, we observe that our model with LCN achieves better performance on these data sets except for mnist-back-rand data set. The results prove that LCN helps conventional PCA performance, which is explained in section 3.1.1. The main reason why LCN performs worse in mnist-back-rand data set is that, the mean and standard deviation in the data set are all corrupted noise information due to large area random noise in the background, which degrades the performance. The LCN boosts the performance in natural images, which is validated by the improvements in MNIST basic, mnist-rotation, mnist-back-image and mnist-back-image-rotation data sets.

4.2 Effect of trans-layer connection

The effect of trans-layer connection is validated on MNIST basic, mnist-rotation, mnist-back-rand, mnist-back-image-rotation data sets in the second experiment. The parameters are set as that of section 4.1. The performance of deep trans-layer PCA network with and without trans-layer connection is recorded in Table 3.

Methods Basic Rot. Bk-rand Bk-im Bk-im-rot
Without trans-layer connection 1.05 6.62 5.94 10.58 34.58
With trans-layer connection 0.98 6.36 5.22 9.95 33.55
Table 3: Comparison of classification error rates (%) on MNIST variations data sets using deep trans-layer PCA network with and without trans-layer connection.

From Table 3, we observe that deep trans-layer PCA networks with trans-layer connection have a consistently better performance than those of without trans-layer connection. Trans-layer connection boosts almost 0.1 percentage performance even on the hardest task of MNIST basic data set, and 1% performance for mnist-back-image-rotation data set. The trans-layer connection in deep trans-layer PCA network provides a more complete representation that is always helpful for recognition.

4.3 Digit recognition on Mnist and Mnist variations data sets

We report the performance of the implemented model on MNIST and MNIST variations data sets compared with other methods such as convolution network (ConvNet) LeCun_ICCV09 and ScatNet-2 Mallat_PAMI13 . The performance of these methods is recorded in Table 4 of MNIST data set and Table 5 of MNIST variations data sets respectively. For fair comparison, the following results do not include the results using augmented samples.

Methods MNIST error rates (%)
K-NN-SCM Shape_PAMI02 0.63
K-NN-IDM Deform_PAMI07 0.54
CDBN Andrew_ICML09 0.82
HSC Yukai_CVPR11 0.77
ConvNet LeCun_ICCV09 0.53
Stochastic Pooling ConvNet Fergus_ICLR13 0.47
Conv. Maxout + Dropout Bengio_ICML13 0.45
ScatNet-2 () Mallat_PAMI13 0.43
PCANet Mayi_PCANet 0.66
Deep Trans-layer PCA Network 0.58 111The matlab code can be downloaded from https://github.com/wentaozhu/Deep-trans-layer-unsupervised-network.git

Deep Trans-layer Autoencoder Network

0.55 111The matlab code can be downloaded from https://github.com/wentaozhu/Deep-trans-layer-unsupervised-network.git
Table 4: MNIST classification error rates (%). Note that the compared methods are not including methods that augment the training data.
Methods Basic Rot. Bk-rand Bk-im Bk-im-rot Rect Rect-im Con.
CAE-2 Bengio_ICML11 2.48 9.66 10.90 15.50 45.23 1.21 21.54 -
TIRBM Lee_ICML12 - 4.20 - - 35.50 - - -
PGBM
+DN-1 Lee_ICML13
- - 6.08 12.25 36.76 - - -
ScatNet-2
() Mallat_PAMI13
1.27 7.48 18.40 12.30 50.48 0.01 8.02 6.50
PCANet Mayi_PCANet 1.06 7.37 6.19 10.95 35.48 0.24 14.00 4.36
Deep Trans-layer
PCA Network
0.98 6.36 5.22 9.95 33.55 0.09 13.18 3.52
Deep Trans-layer
Autoencoder Network
1.02 7.56 5.56 8.37 34.43 0.01 13.01 2.90
Table 5: Comparison of classification error rates (%) on MNIST variations data sets.

From Table 4, the results show that the deep trans-layer unsupervised network is only inferior to ScatNet-2 and enhanced Convolution Network related methods. It is worthy to mention that the performance of ScatNet-2 is achieved by connected with a non-linear SVM with RBF kernels with tuned parameters, but our model is connected with a linear SVM with all default parameters in LIBLINEAR software kit LibLinear . Our model’s performance (0.55) is highly comparable to that of Convolution Network (0.53) on MNIST data set LeCun_ICCV09 . Because MNIST data set contains too many training samples with small intra-class variability, most of methods work well on this data set and the tiny difference is not much meaningful statistically. Despite this, the deep trans-layer PCA network boosts almost 0.1% performance higher than that of PCA related method, PCANet.

From Table 5, we observe that the deep trans-layer unsupervised network achieves the best performance on six data sets with a simple linear SVM classifier. Our model has a highly superior performance than other methods. It is sufficient to prove that the proposed structure of LCN pre-processing and trans-layer connection work well in the convolution structure with unsupervised filters.

4.4 Object recognition on Caltech 101 data set

We also evaluate the model for object recognition task on Caltech 101 data set. Caltech 101 data set contains color images belonging to 102 categories including a background class. The number of each class’s images varies from 31 to 800 LiFeifei_CVPR04 . The pre-processing of the data set is to convert the images into gray scale, and adjust the longer side of the image to 300 with preserved aspect ratio. Two typical tasks are conducted. One is with a training set of 15 samples per class. The other is with training set of 30 samples per class. The training sets are randomly sampled from Caltech 101, and the rest are test set. Five rounds of experiments are recorded, and the performance is recorded as the average of the five rounds of results.

Parameters are set as follows. The filter size is set to pixels, the number of filters is set to , the block size is set to a quarter of the image size and the size of strides is set to the half size of the blocks. The WPCA is used to reduce the dimension of each block’s trans-layer representation from 256 to 64. Linear SVM with default parameters in LIBLINEAR LibLinear is used to tackle the recognition task. Comparison results on gray raw images are recorded in Table 6.

Methods 15 samples per class (%) 30 samples per class (%)
CDBN Andrew_ICML09 57.70 1.50 65.40 0.50
ConvNet LeCun_NIPS10 57.60 0.40 66.30 1.50
DeconvNet Fergus_CVPR10 58.60 0.70 66.90 1.10
Chen et al. Chen_ICML11 58.20 1.20 65.80 0.60
Zou et al. Andrew_NIPS12 - 66.50
HSC Yukai_CVPR11 - 74.0
PCANet Mayi_PCANet 61.46 0.76 68.56 1.01
Deep Trans-layer
PCA Network
67.11 0.64 75.98 0.58
Deep Trans-layer
Autoencoder Network
67.03 0.56 75.90 0.19
Table 6: Comparison of classification accuracy (%) in terms of mean and stand deviation based on gray-level images of Caltech 101 data set.

The Table 6 shows that the deep trans-layer unsupervised network gets the impressive performance trained by 15 samples per class and 30 samples per class tasks respectively by simply using the unsupervised methods. More surprisingly, our model with no elaborately tuned parameters gets about 2% upper than that of HSC Yukai_CVPR11 on 30 samples per class task. The proposed simple network really makes a high progress for the data set.

4.5 Face verification on LFW-a data set

Face verification task is conducted with our model on LFW-a data set Wolf_PAMI11 . The LFW data set contains more than 13,000 faces of 5,749 different individuals, and these images were collected from the web of unconstrained conditions. In the LFW data set, the unsupervised setting is used for the deep trans-layer PCA network for sufficiently validating of representation effectiveness. The LFW-a data set with alignment is used, and we cropped the face images into pixels. The standard evaluation protocol on LFW is followed for performance evaluation. The histogram block size is with non-overlapping. Other parameters are set the same as before. The WPCA is used to reduce the dimension of trans-layer PCA representation to 3,200 after additional square-root operation in the data set. Then use NN classifier with cosine distance to tackle the verification task. The performance is recorded in Table 7 for unsupervised setting with a single descriptor.

Methods Verification accuracy (%)
POEM Caplier_TIP12 82.70 0.59
High-dim. LBP Sunjian_CVPR13 84.08
High-dim. LE Sunjian_CVPR13 84.58
SFRD Cuizhen_CVPR13 84.81
I-LQP Jurie_BMVC12 86.20 0.46
OCLBP Wolf_ICCV13 86.66 0.30
PCANet Mayi_PCANet 86.28 1.14
Deep Trans-layer PCA Network 87.10 0.43
Table 7: Comparison of verification rates (%) on LFW-a data set under unsupervised setting with a single descriptor.

Table 7 shows that the deep trans-layer PCA network achieves the best performance with the unsupervised setting on the LFW-a data set. Also, the boosted performance of our model in contrast to PCANet with the same 3,200 dimensions reveals that, the trans-layer architecture provides uncorrelated information in the increased dimensions, which is beneficial for tasks.

5 Discussion

One of the typical cases to illustrate why the trans-layer representation works is that the local information of naevi in our faces may vanish in the upper layer’s unsupervised representation for the size of receptive fields are increased and the local discriminative information may be neglected. However, the naevi is of highly discriminative features to recognize each person. In conventional deep neural networks, such discriminative details also hard to preserve with the increasing of layer numbers. The trans-layer representation from the bottom layer preserves these details and is helpful for recognition.

The trans-layer representation scheme is quite successful for the convolution network of PCA and auto encoder filters, because the number of filters is relatively too small to preserve enough information for tasks. And information will be lost rapidly with the increase of layer numbers, as illustrated in Fig. 3. If the model uses only the second layer’s feature maps, there would be much noise and lose much useful information. Therefore, the second layer’s feature maps are not enough for classification, and the trans-layer connection is quite helpful.

Figure 3: Illustration of the second layer’s feature maps on MNIST data set, five maps per category. We can hardly recognize most maps for much information loss in the multi-layer PCA network.

The deep trans-layer PCA network also has some problems to be improved despite that LCN and trans-layer representation are added into it. The main problem is the high dimensions of the trans-layer representation. Although good representation for tasks needs high dimensions, we should make the dimensions of representation as low as possible providing good representational ability. So the future work could include that other translation and rotation invariance preserving methods to reduce the dimensions of the trans-layer representations such as pooling operations LeCun_ICCV09 ; LeCun_98 ; Fergus_ICLR13 .

6 Conclusion

In this paper, a novel feature learning and representation model, the deep trans-layer unsupervised network, is demonstrated. Several key elements are added to boost the deep unsupervised learning such as the LCN operation and trans-layer representation. Several experiments on digit recognition and object recognition tasks have shown that, the proposed method based on layer-wise unsupervised learning the local receptive features also obtains the impressive results by trans-layer representation. The deep trans-layer unsupervised network has a quite simple structure with much few parameters, in which only the cascaded local receptive filters need to be learned by unsupervised methods. The structure of trans-layer network also provides a novel direction for us to mine.

References

References

  • (1) D. G. Lowe, Distinctive image features from scale-invariant keypoints, International journal of computer vision 60 (2) (2004) 91–110.
  • (2) N. Dalal, B. Triggs, Histograms of oriented gradients for human detection, in: Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, Vol. 1, IEEE, 2005, pp. 886–893.
  • (3) G. E. Hinton, R. R. Salakhutdinov, Reducing the dimensionality of data with neural networks, Science 313 (5786) (2006) 504–507.
  • (4) T.-H. Chan, K. Jia, S. Gao, J. Lu, Z. Zeng, Y. Ma, Pcanet: A simple deep learning baseline for image classification?, arXiv preprint arXiv:1404.3606.
  • (5) K. Jarrett, K. Kavukcuoglu, M. Ranzato, Y. LeCun, What is the best multi-stage architecture for object recognition?, in: Computer Vision, 2009 IEEE 12th International Conference on, IEEE, 2009, pp. 2146–2153.
  • (6)

    A. Coates, A. Y. Ng, Learning feature representations with k-means, in: Neural Networks: Tricks of the Trade, Springer, 2012, pp. 561–580.

  • (7) L. Fei-Fei, R. Fergus, P. Perona, Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories, Computer Vision and Image Understanding 106 (1) (2007) 59–70.
  • (8) T. Ojala, M. Pietikainen, T. Maenpaa, Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, Pattern Analysis and Machine Intelligence, IEEE Transactions on 24 (7) (2002) 971–987.
  • (9)

    A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in neural information processing systems, 2012, pp. 1097–1105.

  • (10) L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, R. Fergus, Regularization of neural networks using dropconnect, in: Proceedings of the 30th International Conference on Machine Learning (ICML-13), 2013, pp. 1058–1066.
  • (11) Q. V. Le, A. Karpenko, J. Ngiam, A. Y. Ng, Ica with reconstruction cost for efficient overcomplete feature learning, in: Advances in Neural Information Processing Systems, 2011, pp. 1017–1025.
  • (12) J. Bruna, S. Mallat, Invariant scattering convolution networks, Pattern Analysis and Machine Intelligence, IEEE Transactions on 35 (8) (2013) 1872–1886.
  • (13) K. Yu, Y. Lin, J. Lafferty, Learning image representations from the pixel level via hierarchical sparse coding, in: Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, IEEE, 2011, pp. 1713–1720.
  • (14) Y.-L. Boureau, F. Bach, Y. LeCun, J. Ponce, Learning mid-level features for recognition, in: Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, IEEE, 2010, pp. 2559–2566.
  • (15)

    H. Lee, R. Grosse, R. Ranganath, A. Y. Ng, Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations, in: Proceedings of the 26th Annual International Conference on Machine Learning, ACM, 2009, pp. 609–616.

  • (16) R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, C.-J. Lin, Liblinear: A library for large linear classification, The Journal of Machine Learning Research 9 (2008) 1871–1874.
  • (17) H. Larochelle, D. Erhan, A. Courville, J. Bergstra, Y. Bengio, An empirical evaluation of deep architectures on problems with many factors of variation, in: Proceedings of the 24th international conference on Machine learning, ACM, 2007, pp. 473–480.
  • (18) Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86 (11) (1998) 2278–2324.
  • (19) S. Belongie, J. Malik, J. Puzicha, Shape matching and object recognition using shape contexts, Pattern Analysis and Machine Intelligence, IEEE Transactions on 24 (4) (2002) 509–522.
  • (20) D. Keysers, T. Deselaers, C. Gollan, H. Ney, Deformation models for image recognition, Pattern Analysis and Machine Intelligence, IEEE Transactions on 29 (8) (2007) 1422–1435.
  • (21) M. D. Zeiler, R. Fergus, Stochastic pooling for regularization of deep convolutional neural networks, arXiv preprint arXiv:1301.3557.
  • (22) I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, Y. Bengio, Maxout networks, arXiv preprint arXiv:1302.4389.
  • (23) S. Rifai, P. Vincent, X. Muller, X. Glorot, Y. Bengio, Contractive auto-encoders: Explicit invariance during feature extraction, in: Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011, pp. 833–840.
  • (24) K. Sohn, H. Lee, Learning invariant representations with local transformations, arXiv preprint arXiv:1206.6418.
  • (25) K. Sohn, G. Zhou, C. Lee, H. Lee, Learning and selecting features jointly with point-wise gated B oltzmann machines, in: Proceedings of The 30th International Conference on Machine Learning, 2013, pp. 217–225.
  • (26) K. Kavukcuoglu, P. Sermanet, Y.-L. Boureau, K. Gregor, M. Mathieu, Y. L. Cun, Learning convolutional feature hierarchies for visual recognition, in: Advances in neural information processing systems, 2010, pp. 1090–1098.
  • (27) M. D. Zeiler, D. Krishnan, G. W. Taylor, R. Fergus, Deconvolutional networks, in: Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, IEEE, 2010, pp. 2528–2535.
  • (28) B. Chen, G. Polatkan, G. Sapiro, L. Carin, D. B. Dunson, The hierarchical beta process for convolutional factor analysis and deep learning, in: Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011, pp. 361–368.
  • (29) W. Zou, S. Zhu, K. Yu, A. Y. Ng, Deep learning of invariant features via simulated fixations in video, in: Advances in Neural Information Processing Systems, 2012, pp. 3212–3220.
  • (30) L. Wolf, T. Hassner, Y. Taigman, Effective unconstrained face recognition by combining multiple descriptors and learned background statistics, Pattern Analysis and Machine Intelligence, IEEE Transactions on 33 (10) (2011) 1978–1990.
  • (31) N.-S. Vu, A. Caplier, Enhanced patterns of oriented edge magnitudes for face recognition and image matching, Image Processing, IEEE Transactions on 21 (3) (2012) 1352–1365.
  • (32) D. Chen, X. Cao, F. Wen, J. Sun, Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification, in: Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, IEEE, 2013, pp. 3025–3032.
  • (33) Z. Cui, W. Li, D. Xu, S. Shan, X. Chen, Fusing robust face region descriptors via multiple metric learning for face recognition in the wild, in: Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, IEEE, 2013, pp. 3554–3561.
  • (34) S. U. Hussain, T. Napoléon, F. Jurie, et al., Face recognition using local quantized patterns, in: British Machive Vision Conference, 2012.
  • (35) O. Barkan, J. Weill, L. Wolf, H. Aronowitz, Fast high dimensional vector multiplication face recognition, in: Computer Vision (ICCV), 2013 IEEE International Conference on, IEEE, 2013, pp. 1960–1967.