1 Introduction
The development of deep neural networks has achieved remarkable advancement in the field of machine learning during the past decade. By constructing a hierarchical or "deep" structure, the model is able to learn good representations from raw data in both supervised or unsupervised settings which is believed to be its key ingredient of success. Successful application areas include computer vision, speech recognition, natural language processing and more
(Goodfellow:Bengio2016, ).Currently, almost all the deep neural networks use backpropagation (bp, ; rumelhart1986learning, )
with stochastic gradient descent as the workhorse behind the scene for updating parameters during training. Indeed, when the model is composed of differentiable components (e.g., weighted sum with nonlinear activation functions), it appears that backprop is still currently the best choice. Some other methods such as target propagation
(targetprop, ) has been proposed as an alternative way of training the neural networks, the effectiveness and popularity are however still in a premature stage. For instance, the work in dtp proved that target propagation can be at most as good as backprop, and in practice an additional backpropagation for finetuning is often needed. In other words, the goodold backpropagation is still the most effective way to train a differentiable learning system such as neural networks.On the other hand, the need to explore the possibility to build a multilayered or deep model using nondifferentiable modules is not only of academic interest but also with important application potentials. For instance, treebased ensembles such as Random Forest
(Breiman2001, ) or gradient boosting decision trees (GBDTs) (gbdt, )are still the dominant way of modeling discrete or tabular data in a variety of areas, it thus would be of great interest to obtain a hierarchical distributed representation learned by tree ensembles on such data. In such cases, since there is no chance to use chain rule to propagate errors, backpropagation is no longer possible. This yields to two fundamental questions: First, can we construct a multilayered model with nondifferentiable components, such that the outputs in the intermediate layers can be regarded as a distributed representation? Second, if so, how to jointly train such models without the help of backpropagation? The goal of this paper is to provide such an attempt.
Recently Zhou and Feng gcForest17 proposed the Deep Forest framework, which is the first attempt of constructing a multilayered model using tree ensembles. Concretely, by introducing finegrained scanning and cascading operations, the model is able to construct a multilayered structure with adaptive model complexity and achieved competitive performance across a board range of tasks. The gcForest model proposed in (gcForest17, )
utilized all strategies for diversity enhancement of ensemble learning, however, this approach is only suitable in a supervised learning setting. Meanwhile, it is still not clear how to construct a multilayered model by forest that
explicitly examine its representation learning ability. Such explorations for representation learning should be made since many previous researches have suggested that, a multilayered distributed representations (Hinton:1986:DR:104279.104287, ) may be the key reason for the success of deep neural networks (bengio2013representation, ).In this work, we aim to take the best parts of both worlds: the excellent performance of tree ensembles and the expressive power of hierarchical distributed representations (which has been mainly explored in neural networks). Concretely, we propose the first multilayered structure using gradient boosting decision trees as building blocks per layer with an explicit emphasis on its representation learning ability and the training procedure can be jointly optimized via a variant of target propagation. The model can be trained in both supervised and unsupervised settings. This is the first demonstration that we can indeed obtain hierarchical and distributed representations using trees which was commonly believed only possible for neural networks or differentiable systems in general. Theoretical justifications as well as experimental results showed the effectiveness of this approach.
The rest of the paper is organized as follows: first, some more related works are discussed; second, the proposed method with theoretical justifications are presented; finally, experiments and conclusions are illustrated and discussed.
2 Related Works
There is still no universal theory in explaining why a deep model works better than a shallow one. Many of the current attempts (bengio2013better, ; bengio2013generalized, ) for this question are based on the conjecture that it is the hierarchical distributed representations learned from data are the driven forces behind the effectiveness of deep models. Similar works such as (bengio2013better, )
conjectured that better representations can be exploited to produce fastermixing Markov chains, therefore, a deeper model always helps. Tishby and Zaslavsky
TishbyZ15 treated the hidden layers as a successive refinement of relevant information and a deeper structure helps to speed up such process exponentially. Nevertheless, it seems for a deep model to work well, it is critical to obtain a better feature rerepresentation from intermediate layers.For a multilayered deep model with differentiable components, backpropagation is still the dominant way for training. In recent years, some alternatives have been proposed. For instance, targetpropagation (targetprop, ) and difference target propagation (dtp, )
propagate the targets instead of errors via the inverse mapping. By doing so, it helps to solve the vanishing gradient problem and the authors claim it is a more biologically plausible training procedure. Similar approaches such as feedbackalignment
(feedback, ) used asymmetric feedback connections during training and direct feedback alignment (directFeedback, ) showed it is possible when the feedback path is disconnected from the forward path. Currently, all these alternatives stay in the differentiable regime and their theoretical justifications depend heavily on calculating the Jacobians for the activation functions.Ensemble learning (Zhou2012, ) is a powerful learning paradigm which often uses decision trees as its base learners. Bagging (Breiman1996, ) and boosting (Freund:Schapire:1999, ) , for instance, are the driven forces of Random Forest (Breiman2001, ) and gradient boosting decision trees (gbdt, )
, respectively. In addition, some efficient implementations for GBDTs such as XGBoost
(chen2016xgboost, )has become the best choice for many industrial applications and data science projects, ranging from predicting clicks on Ads
(DBLP:conf/kdd/HePJXLXSAHBC14, ), to discovering Higgs Boson (chen2015higgs, ) and numerous data science competitions in Kaggle^{1}^{1}1www.kaggle.com and beyond. Some more recent works such as eForest (eForest, ) showed the possibility to recover the input pattern with almost perfect reconstruction accuracy by forest. Due to the unique property of decision trees, such models are naturally suitable for modeling discrete data or datasets with mixedtypes of attributes.3 The Proposed Method
Consider a multilayered feedforward structure with intermediate layers and one final output layer. Denote where as the output for each layer including the input layer and the output layer . For a particular input data , the corresponding output at each layer is in , where . The learning task is therefore to learn the mappings for each layer , such that the final output minimize the empirical loss on training set. Mean squared errors or crossentropy with extra regularization terms are some common choices for the loss . In an unsupervised setting, the desired output
can be the training data itself, which leads to an autoencoder and the loss function is the reconstruction errors between the output and the original input.
When each is parametric and differentiable, such learning task can be achieved in an efficient way using backpropagation. The basic routine is to calculate the gradients of the loss function with respect to each parameter at each layer using the chain rule, and then perform gradient descent for parameter updates. Once the training is done, the output for the intermediate layers can be regarded as the new representation learned by the model. Such hierarchical dense representation can be interpreted as a multilayered abstraction of the original input and is believed to be critical for the success of deep models.
However, when is nondifferentiable or even nonparametric, backprop is no longer applicable since calculating the derivative of loss function with respect to its parameters is impossible. The rest of this section will focus on solving this problem when are gradient boosting decision trees.
First, at iteration , assume the obtained from the previous iteration are given, we need to obtain an "pseudoinverse" mapping paired with each such that . This can be achieved by minimizing the expected value of the reconstruction loss function as: ,where the loss
can be the reconstruction loss. Like an autoencoder, random noise injection is often suggested, that is, instead of using a pure reconstruction error measure, it is good practice setting
as: . By doing so, the model is more robust in the sense that the inverse mapping is forced to learn how to map the neighboring training data to the right manifold. In addition, such randomness injection also helps to design a generative model by treating the inverse mapping direction as a generative path which can be considered as future works for exploration.Second, once we updated , we can use it as given and update the forward mapping for the previous layer . The key here is to assign a pseudolabels for where , and each layer’s pseudolabel is defined to be . That is, at iteration , for all the intermediate layers, the pseudolabels for each layer can be "aligned" and propagated from the top to bottom. Then, once the pseudolabels for each layer is computed, each can follow a gradient ascent step towards the pseudoresiduals just like a typical regression GBDT.
The only thing remains is to set the pseudolabel for the final layer to make the whole structure ready for update. It turns out to be easy since at layer , one can always use the real labels when defining the top layer’s pseudolabel. For instance, it is natural to define the pseudolabel of the top layer as: . Then, is set to fit towards the pseudoresiduals . In other words, at iteration , the top layer compute its pseudolabel and then produce the pseudolabels for all the other layer via the inverse functions, then each can thus be updated accordingly. Once all the get updated, the procedure can then move to the next iteration to update . In practice, a bottom up update is suggested (update before for ) and each can go several rounds of additive boosting steps towards its current pseudolabel.
When training a neural network, the initialization can be achieved by assigning random Gaussian noise to each parameter, then the procedure can move on to the next stage for parameter update. For treestructured model described here, it is not a trivial task to draw a random tree structure from the distribution of all the possible tree configurations, therefore instead of initializing the tree structure at random, we produce some Gaussian noise to be the output of intermediate layers and train some very tiny trees to obtain , where index denote the tree structures obtained in this initialization stage. Then the training procedure can move on to iterative update forward mappings and inverse mappings. The whole procedure is summarized in Algorithm 1 and illustrated in Figure 1.
It is worth noticing that the work in xgboostGPU utilized GPUs to speed up the time required to train a GBDT and Korlakai and Ran xgbDropout showed an efficient way of conducting dropout techniques for GBDTs which will give a performance boost further. For a multidimensional output problem, the naïve approaches using GBDTs would be memory inefficient. Si et al. fastxgboost proposed an efficient way of solving such problem which can reduce the memory by an order of magnitude in practice.
In classification tasks, one could set the forward mapping in the top layer as a linear classifier. There are two main reasons of doing this: First, by doing so, the lower layers will be forced to learn a feature rerepresentation that is as linear separable as possible which is a useful property to have. Second, often the difference of the dimensionality between the output layer and the layer below is big, as a result, an accurate inverse mapping may be hard to learn. When using a linear classifier as the forward mapping at the top layer, there is no need to calculate that particular corresponding inverse mapping since the pseudolabel for the layer below can be calculated by taking the gradient of the global loss with respect to the output of the last hidden layer.
A similar procedure such as target propagation (targetprop, ) has been proposed to use the interlayer feedback mappings to train a neural network. They proved that under certain conditions, the angle between the update directions for the parameters of the forward mappings and the update directions when trained with backpropagation is less than 90 degree. However, the proof relies heavily on the computing the Jacobians of and , therefore, their results are only suitable for neural networks.
The following theorem proves that, under certain conditions, an update in the intermediate layer towards its pseudolabel helps to reduce the loss of the layer above, and thus helps to reduce the global loss. The proof here does not rely on the differentiable property of and .
Theorem 1. Denote an update of to moves its output from to , where and are in and denote the input for as which is in . Assume each and preserve the isometry on its neighbors. Now suppose such update for reduced its local loss, that is, , then it helps to reduce the loss for the layer above, that is, the following holds:
Proof. By assumption, we have the following:
To conclude this section, here we discuss several reasons for the need to explore nondifferential components in designing multilayered models. Firstly, current adversarial attacks (foolCNN, ; adversarial17, ) are all based on calculating the derivative of the final loss with respect to the input. That is, regardless of the training procedure, one can always attack the system as long as chain rule is applicable. Nondifferentiable modules such as trees can naturally block such calculation, therefore, it would more difficult to perform malicious attacks. Secondly, there are still numerous datasets of interest that are best suitable to be modeled by trees. It would be of great interests and potentials to come up with algorithms that can blend the performance of tree ensembles with the benefit of having a multilayered representation.
4 Experiments
The experiments for this section is mainly designed to empirically examine if it is feasible to jointly train the multilayered structure proposed by this work. That is, we make no claims that the current structure can outperform CNNs in computer vision tasks. More specifically, we aim to examine the following questions: (Q1) Does the training procedure empirically converge? (Q2) What does the learned features look like? (Q3) Does depth help to learn a better representation? (Q4) Given the same structure, what is the performance compared with neural networks trained by either backpropagation or targetpropagation?
With the above questions in mind, we conducted 3 sets of experiments with both synthetic data and realworld applications which results are presented below.
4.1 Synthetic Data
As a sanity check, here we train two small multilayered GBDTs on synthetic datasets.
We generated points with 2 classes (70 % for training and 30 % for testing) on as illustrated in Figure 1(a). The structure we used for training is where the input points are in and the output is a 0/1 classification prediction. The mGBDT used in both forward and inverse mappings have a maximum depth of 5 per tree with learning rate of 0.1. The output of the last hidden layer (which is in ) is visualized in Figure 1(b). Clearly, the model is able to transform the data points that is easier to separate.
We also conducted an unsupervised learning task for autoencoding.
points in are generated, as shown in Figure 2(a). Then we built an autoencoder using mGBDTs with structure with MSE as its reconstruction loss. That is, the model is forced to learn a mapping from to , then maps it back to the original space with low reconstruction error as objective. The reconstructed output is presented in Figure 2(b). The encodings for the input points are impossible to visualize directly, here we use a common strategy to visualize some pairs of dimensions for the encodings in as illustrated in Figure 4. The representation for the points is indeed a distributed representation (Hinton:1986:DR:104279.104287, ) as some of the dimension captures the curvature whereas others preserve the relative distance among points.4.2 Income Prediction
The income prediction dataset (Lichman2013, ) consists of samples ( for training and
for testing) of tabular data with both categorical and continuous attributes. Each sample consists of a person’s social background such as race, sex, workclass, etc. The task here is to predict whether this person makes over 50K a year. Onehot encoding for the categorical attributes make each training data in
. The multilayered GBDT structure we used is. Gaussian noise with zero mean and standard deviation of 0.3 is injected in
. To avoid training the inverse mapping on the top layer, we set the top classification layer to be a linear with crossentropy loss, other layers all use GBDTs for for forward/inverse mappings with the same hyperparameters in section 4.1. The learning rate at top layer is determined by crossvalidation. The output for each intermediate layers are visualized via TSNE (tSNE, ) in Figure 5.For a comparison, we also trained the exact same structure on neural networks using the target propagation and standard backpropagation , respectively. Adam (kingma2014adam, )
with a learning rate of 0.001 and ReLU activation are used for both cases. Dropout rate of 0.25 is used for backprop. A vanilla XGBoost with 100 additive trees with a maximum depth of 7 per tree is also trained for comparison, the learning rate for XGBoost is set to be 0.3. Finally, we stacked 3 XGBoost of the same configurations as the vanilla XGBoost and used one additional XGBoost as the second stage of stacking, 3fold validation is used which is a common request to perform stacking. More stacking levels will produce severe overfitting results and are not included here.
Income Dataset  Protein Dataset  

XGBoost  .8719  .5937 .0324 
XGBoost Stacking  .8697  .5592 .0400 
.8491  .5756 .0465  
.8534  .5907 .0268  
Multilayered GBDT  .8742  .5948 .0268 
Experimental results are summarized in Figure 6 and Table 1. First, multilayered GBDT forest (mGBDT) achieved the highest accuracy compared to DNN approaches trained by either backprop or targetprop, given the same model structure. It also performs better than single GBDTs or stacking multiple ones in terms of accuracy. Second, converges not as good as as expected (a consistent result with dtp ), whereas the same structure using GBDT layers can achieve a lower training loss without overfitting.
4.3 Protein Localization
The protein dataset (Lichman2013, ) is a 10 class classification task consists of only 1484 training data where each of the 8 input attributes is one measurement of the protein sequence, the goal is to predict protein localization sites with 10 possible choices. 10fold crossvalidation is used for model evaluation since there is no test set provided. We trained a multilayered GBDT using structure . Due to the robustness of tree ensembles, all the training hyperparameters are the same as we used in the previous section. Likewise, we trained two neural networks and with the same structure, and the training parameters were determined by crossvalidation for a fair comparison. Experimental results are presented in Table 1. Again mGBDT achieved best performance among all. XGBoost Stacking had a worse accuracy than using a single XGBoost, this is mainly because overfitting has occurred. We also visualized the output for each mGBDT layer using TSNE in Figure 7. It can be shown that the quality of the representation does get improved with model depth.
The training and testing curves for 10fold crossvalidation are plotted with mean value in Figure 8. The multilayered GBDT (mGBDT) approach converges much faster than NN approaches, as illustrated in Figure 7(a). Only 50 epoch is needed for mGBDT whereas NNs require 200 epochs for both backprop and targetprop scenarios. In addition, is still suboptimal than and mGBDT achieved highest accuracy among all. We also examined the effect when we vary the number of intermediate layers on protein datasets. To make the experiments manageable, the dimension for each intermediate layer is fixed to be 16. The results are summarized in Table 2. It can be shown that mGBDT is more robust compared with as we increase the intermediate layers. Indeed, the performance dropped from to when using targetprop for neural networks whereas mGBDT can still perform well when adding extra layers.
Model Structure  

8>10  .5873 .0396  N/A  .5937 .0324 
8>16>10  .5803 .0316  .5964 .0343  .6160 .0323 
8>16>16>10  .5907 .0268  .5756 .0465  .5948 .0268 
8>16>16>16>10  .5901 .0270  .4759 .0429  .5897 .0312 
8>16>16>16>16>10  .5768 .0286  .3654 .0452  .5782 .0229 
5 Conclusion and Future Explorations
In this paper, we present a novel multilayered GBDT forest (mGBDT) with explicit representation learning ability that can be jointly trained with a variant of target propagation. Due to the excellent performance of tree ensembles, this approach is of great potentials in many application areas where neural networks are not the best fit. The work also showed that, to obtain a multilayered distributed representations is not tired to differentiable systems. Theoretical justifications, as well as experimental results confirmed the effectiveness of this approach. Here we list some aspects for future explorations.
Deep Forest Integration. One important feature of the deep forest model proposed in (gcForest17, ) is that the model complexity can be adaptively determined according to the input data. Therefore, it is interesting to integrating several mGBDT layers as feature extractor into the deep forest structure to make the system not only capable of learning representations but also can automatically determine its model complexity.
Structural Variants and Hybird DNN. A recurrent or even convolutional structure using mGBDT layers as building blocks are now possible since the training method does not making restrictions on such structural priors. Some more radical design is possible. For instance, one can embed the mGBDT forest as one or several layers into any complex differentiable system and use mGBDT layers to handle tasks that are best suitable for trees. The whole system can be jointly trained with a mixture of different training methods across different layers. Nevertheless, there are plenty of room for future explorations.
References
 [1] I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, Cambridge, MA, 2016.
 [2] P. Werbos. Beyond regression: New tools for prediction and analysis in the behavioral sciences. PhD thesis, Harvard University, 1974.
 [3] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by backpropagating errors. Nature, 323(6088):533, 1986.
 [4] Y. Bengio. How autoencoders could provide credit assignment in deep networks via target propagation. arXiv:1407.7906, 2014.
 [5] D.H. Lee, S. Zhang, A. Fischer, and Y. Bengio. Difference target propagation. In ECML PKDD, pages 498–515, 2015.
 [6] L. Breiman. Random forests. Machine Learning, 45(1):5–32, 2001.
 [7] J. H. Friedman. Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29:1189–1232, 2000.
 [8] Z.H. Zhou and J. Feng. Deep forest: Towards an alternative to deep neural networks. In IJCAI, pages 3553–3559, 2017.
 [9] G. E. Hinton, J. L. McClelland, and D. E. Rumelhart. Distributed representations. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1, pages 77–109. 1986.
 [10] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE Trans. Pattern Analysis and Machine Intelligence, 35(8):1798–1828, 2013.
 [11] Y. Bengio, G. Mesnil, Y. Dauphin, and S. Rifai. Better mixing via deep representations. In ICML, pages 552–560, 2013.
 [12] Y. Bengio, L. Yao, G. Alain, and P. Vincent. Generalized denoising autoencoders as generative models. In NIPS, pages 899–907, 2013.
 [13] N. Tishby and N. Zaslavsky. Deep learning and the information bottleneck principle. arXiv:1503.02406, 2015.

[14]
T. P. Lillicrap, D. Cownden, D. B. Tweed, and C. J. Akerman.
Random synaptic feedback weights support error backpropagation for deep learning.
Nature communications, 7:13276, 2016.  [15] A. Nøkland. Direct feedback alignment provides learning in deep neural networks. In NIPS, pages 1037–1045, 2016.
 [16] Z.H. Zhou. Ensemble Methods: Foundations and Algorithms. CRC, Boca Raton, FL, 2012.
 [17] L. Breiman. Bagging predictors. Machine Learning, 24(2):123–140, 1996.

[18]
Y. Freund and R. E. Schapire.
A short introduction to boosting.
Journal of Japanese Society for Artificial Intelligence
, 14(5):771–780, 1999.  [19] T.Q. Chen and C. Guestrin. XGBoost: A scalable tree boosting system. In KDD, pages 785–794, 2016.
 [20] X. He, J. Pan, O. Jin, T. Xu, B. Liu, T. Xu, Y. Shi, A. Atallah, R. Herbrich, S. Bowers, and J. Quiñonero. Practical lessons from predicting clicks on ads at facebook. In ADKDD, pages 5:1–5:9, 2014.
 [21] T.Q. Chen and T. He. Higgs boson discovery with boosted trees. In NIPS Workshop, pages 69–80, 2015.
 [22] J. Feng and Z.H. Zhou. Autoencoder by forest. In AAAI, 2018.
 [23] M. Rory and F. Eibe. Accelerating the xgboost algorithm using GPU computing. PeerJ Computer Science, 3:127, 2017.
 [24] V. R. Korlakai and G. B. Ran. DART: dropouts meet multiple additive regression trees. In AISTATS, pages 489–497, 2015.
 [25] S. Si, H. Zhang, S. S. Keerthi, D. Mahajan, I. S. Dhillon, and C.J. Hsieh. Gradient boosted decision trees for high dimensional sparse output. In ICML, pages 3182–3190, 2017.
 [26] A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. pages 427–436, 2015.
 [27] S. H. Huang, N. Papernot, I. Goodfellow, Y. Duan, and P. Abbeel. Adversarial attacks on neural network policies. arXiv:1702.02284, 2017.
 [28] M. Lichman. UCI machine learning repository, 2013.

[29]
L.J.P van der Maaten and G.E. Hinton.
Visualizing highdimensional data using tsne.
Journal of Machine Learning Research, 9:2579–2605, 2008.  [30] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.
Comments
There are no comments yet.