Relating Cascaded Random Forests to Deep Convolutional Neural Networks for Semantic Segmentation

1 Introduction

A central challenge in computer vision is the assignment of a semantic class label to every pixel in an image, a task known as semantic segmentation. A common strategy for semantic segmentation is to use pixel-level classifiers such as Random Forests (RF)

[4], which have the advantage of being easy to train and performing well on a wide range of tasks, even in the face of little training data. The use of stacked classifiers, such as in Auto-context [32], has been shown to improve performance on many tasks such as object-class segmentation [29], facade segmentation [13], and brain segmentation [32]. However, this strategy has the limitation that the individual classifiers are trained greedily.

Recently, numerous groups have explored the use of Convolutional Neural Networks (CNNs) for semantic segmentation [7, 19, 6, 34], which has the advantage that it enables “end-to-end learning” of all model parameters. This trend is largely inspired by the success of deep CNNs on high-level computer vision tasks, such as image classification [16] and object detection [12]. However, training a deep CNN requires substantial experience and large amounts of labeled data, or availability of a pre-trained CNN for a similar task [2, 3]. Thus, there currently exists a divide between stacked classifiers and deep CNNs.

We propose an alternative solution, exploiting the fundamental connection between decision trees (DT) and NNs

[27] to bridge the gap between stacked classifiers and deep CNNs. This provides a novel approach with the strengths of stacked classifiers, namely robustness to limited training data, and the end-end-learning capacity of NNs. Figure 1 depicts our proposed pipeline.

Contributions. We make the following contributions:

1. We show that a stacked RF with contextual features is a special case of a deep CNN with sparse convolutional kernels. We apply this successfully to semantic segmentation.

2. We describe an exact mapping of a stacked RF to our sparse, deep CNN. We utilize this mapping to initialize the CNN from a greedily trained stacked RF. This is important in the case of limited training samples. We show that this leads to superior results compared to alternative strategies.

3. We describe an approximate mapping of our sparse, deep CNN back to a stacked RF. We show that this improves the performance of a greedily trained stacked RF.

4. Due to our special CNN architecture we are able to gain new insights of the activation pattern of internal layers, with respect to semantic labels. In particular, we observe that the common smoothing strategy in stacked RFs is naturally learned by our CNN.

Figure 1: Overview. Our method (left) and corresponding results (right) for semantic segmentation of somites in microscopy images of developing zebrafish. (1) A stacked RF is trained from an input filter stack, to predict dense semantic labels. (2) The stacked RF is then mapped to a deep CNN and further trained by back-propagation to improve performance. (3) Optionally, the CNN is mapped back to a stacked RF with updated parameters, for improved speed at test time. The new stacked RF performs worse than the CNN, but better than the original RF. Note that the resulting label maps (right, 1-3) correspond to the individual models (left, 1-3). The result images are zoomed-in with respect to the original image at the far left.

2 Related Work

Our work relates to (i) global optimization of RF classifiers, (ii) mapping RF classifiers to neural networks, (iii) feature learning in stacked RF models, (iv) applying CNNs to the task of semantic segmentation, and (v) training CNNs with limited labeled data. We cover these areas in turn.

Global Optimization of RFs. The limitations of traditional greedy RF construction [4] have been addressed by numerous works. In [31]

, the authors learn a DT by the standard greedy construction, followed by a process they call “fuzzification”, replacing all threshold split decisions with smooth sigmoid functions that they interpret as partial or “fuzzy” inheritance by the daughter nodes. They develop a back-propagation algorithm, which begins in the leaves and propagates up one layer at time to the root node, re-optimizing all split parameters of the DT. In

[23]

, they learn to combine the predictions from each DT so that the complementary information between multiple trees is optimally exploited. They identify a suitable loss function, and after training a standard RF, they retrain the distributions stored in the leaves, and prune the DTs to accomplish compression and avoid overfitting. However,

[23] does not retrain the parameters of the internal split nodes of individual DTs, whereas [31] does not retrain the combination of trees in the forest. Conceptually, our approach does both.

Mapping RFs to NNs. In both [31] and [23], RFs were initially trained in a greedy fashion, and then later refined. An alternative but related approach is to map the greedily trained RF to an NN with two hidden layers, and use this as a smart initialization for subsequent parameter refinement by back-propagation [27, 33]. This effectively “fuzzifies” threshold split decisions, and simultaneously enables training with respect to a final loss function on the output of the NN. Hence as opposed to [31] and [23], all model parameters are learned simultaneously in an end-to-end fashion. Additional advantages are that (i) back-propagation has been widely studied in this form, and (ii) back-propagation is highly parallelized, and only needs to propagate over 2 hidden layers, compared to all tree levels as in [31].

Our work builds upon [27, 33]: We extend their approach to a deep CNN, inspired by the Auto-context algorithm [32], for the purpose of semantic segmentation. Furthermore, we propose an approximate algorithm for mapping the trained CNN back to a RF with axis-aligned threshold split functions, for fast inference at test time.

Feature Learning in a RF Framework. The Auto-context algorithm [32] attempts to capture pixel interdependencies in the learning process by iteratively learning a pixel-wise classifier, using the prediction of nearby pixels from the previous iteration as features. This process is closely related to feature learning, due to the introduction of new features during the learning process. Numerous works have generalized the initial approach of Auto-context. In Entangled Random Forests (ERFs) [20], spatial dependencies are captured by “entanglement features” in each DT, without the need for stacking. Geodesic Forests [15] additionally introduce image-aware geodesic smoothing to the class distributions, to be used as features by deeper nodes in the DT. However, despite the fact that ERFs use a soft sigmoid split function to obtain max-margin behaviour with a small number of trees, these approaches are still limited by greedy parameter optimization.

In a more traditional approach to feature learning, Neural Decision Forests [5]

mix RFs and NNs by using multi-layer perceptrons (MLP) as soft split functions, to jointly tackle the problem of data representation and discriminative learning. This approach can obtain superior results with smaller trees, at the cost of more complicated split functions; however, the MLPs in each split node are trained independently of each other. This limitation is addressed in

[14], which trains the entire system end-to-end. However, they adopt a mixed framework, with both differentiable RFs and CNNs, that are trained in an alternating fashion, and applied to image classification. In contrast, we map to the CNN framework, which enables optimization with popular back-propagation algorithm, and apply to the task of semantic segmentation.

CNNs for Semantic Segmentation. While CNNs have proven very successful for high-level vision tasks, such as image classification, they are less popular for the task of dense semantic segmentation, due to their in-built spatial invariance. CNNs can be applied in a tile-based manner [7]; however, this leads to pixel-independent predictions, which require additional measures to ensure spatial consistency [11, 21]. In [19], the authors extend the tile-based approach to “whole-image-at-a-time” processing, in their Fully Convolutional Network (FCN). They address the coarse-graining effect of the CNN by upsampling the feature maps in deconvolution layers, and combining fine-grained and coarse-grained features during prediction. This approach, combining down-sampling with subsequent up-sampling, is necessary to maintain a large receptive field without increasing the size of the convolution kernels, which otherwise become difficult to learn. A variant of FCN called U-Net was recently proposed in [25]. In [6], they minimize coarse-graining by skipping multiple sub-sampling layers and avoid introducing additional parameters by using sparse convolutional kernels in the layers with large receptive fields. They additionally post-process by a fully connected CRF. In [34]

, they address coarse-graining by expressing mean-field inference in a dense CRF as a Recurrent Neural Network (RNN), and concatenating this RNN behind a FCN, for end-to-end training of all parameters. Notably, they demonstrate a significant boost in performance on the Pascal VOC 2012 segmentation benchmark.

In our work we propose a new CNN architecture for semantic segmentation. Contrary to the previous approaches, we avoid coarse-graining effects, which arise in large part due to pre-training a CNN for image classification

on data provided by the ImageNet Large Scale Visual Recognition Challenge (ILSVRC). Instead, we pre-train a stacked RF on a small set of densely labeled data. Our approach is related to the use of sparse kernels in

[6]

; however, we learn the non-zero element(s) of very sparse convolutional kernels during greedy construction of an RF stack. One advantage of this approach is that since the kernels have a very large receptive field, we do not need max-pooling and deconvolution layers, as

e.g.in the FCN. Additionally, in our approach the sparsity of the kernels can be specified by the number of features used in each RF split node, independently of the size of the receptive field.

Training CNNs with Limited Labelled Data. CNNs provide a powerful tool for feature learning; however, their performance relies on a large set of labeled training data. Unsupervised pre-training has been used successfully to leverage small labeled training sets [26, 22]

; however, fully supervised training on large data sets still gives higher performance. Alternatively, transfer learning makes use of

e.g., pseudo-tasks [1], or surrogate training data [9].

More recent practice is to train a CNN on a large training set, and then fine tune the parameters on the target data [12]. However, this requires a closely related task with a large labeled data set, such as ILSVRC. Another strategy to address the dependency on training data, is to expand a small labeled training set through data augmentation [25]. Alternatively, one can use companion objective functions at each hidden layer, as a form of regularization during training [18, 17]. However, this may in principle interfere with the deep network’s ability to learn the optimal internal representations, as noted by the authors.

We propose a novel strategy for addressing the challenge of training deep CNNs given limited training data. Similar in spirit to [10, 18], we employ greedy supervised pre-training, yet in a complementary model, namely the popular Auto-context model. We then map the resulting Auto-context model onto a deep CNN, and refine all weights using back-propagation.

3 Method

In Section 3.1, we review the algorithm for mapping an RF onto an NN with two hidden layers [27, 33]. In Section 3.2, we introduce the relationship between RFs with contextual features and CNNs. In Section 3.3, we describe our main contribution, namely how to map a stack of RFs onto a deep CNN. In Section 3.4, we describe our second contribution, namely an algorithm for mapping our deep CNN back onto the original RF stack, with updated parameters.

3.1 Mapping a RF to a NN with Two Hidden Layers

In the following, we review the existing works [27, 33]. A decision tree consists of a set of split nodes, $n \in N^{S p l i t}$ , and leaf nodes, $l \in N^{L e a f}$ . Each split node $n$ processes the subset $X_{n}$ of the feature space $X$ that reaches it. Usually, $X = R^{F}$ , where $F$ is the number of features. Let $c l (n)$ and $c r (n)$ denote the left and right child node of a split node $n$ . A split node $n$ partitions the set $X_{n}$ into two sets $X_{c l (n)}$ and $X_{c r (n)}$ by means of a split decision. For DTs using axis-aligned split decisions, the split is performed on the basis of a single feature whose index we denote by $f (n)$ , and a respective threshold denoted as $θ (n)$ : $\forall x \in X_{n} : x \in X_{c l (n)} ⟺ x_{f (n)} < θ (n)$ .

For each leaf node $l$ , there exists a unique path from root node $n_{0}$ to leaf $l$ , $P (l) = {n_{i}}_{i = 0}^{d}$ , with $n_{0} . . . n_{d} \in N^{S p l i t}$ and $X_{l} \subseteq X_{n_{d}} \subseteq . . . \subseteq X_{n_{0}}$ . Thus, leaf membership can be expressed as follows:

x \in X_{l} ⟺ \forall n \in P (l) : {\begin{matrix} x_{f (n)} < θ (n) & if X_{l} \subseteq X_{c l (n)} . x_{f (n)} \geq θ (n) & if X_{l} \subseteq X_{c r (n)} . \end{matrix}

(1)

Each leaf node $l$ stores votes for the semantic class labels, $y^{l} = (y_{1}^{l} . . . y_{C}^{l})$ , where $C$

is the number of classes. For a feature vector

x

, we denote the unique leaf of the tree that has

x \in X_{l}

leaf (x)

. The prediction of a DT for feature vector

x

to be of class

c

is given by:

p (c | x) = \frac{y_{c}^{leaf (x)}}{\sum_{c = 1}^{C} y_{c}^{leaf (x)}}

(2)

Using this notation, we now describe how to map a DT to a feed-forward NN, with two hidden layers. Conceptually, the NN separates the task of evaluating the split nodes and evaluating leaf membership into the first and second hidden layers, respectively. See Figure 2 for a sketch of the following description.

Figure 2: Mapping from a RF to a NN. (a) A shallow DT. Nodes are labeled to show mapping to NN. (b) Corresponding NN with two hidden layers. The first hidden layer is connected to the input layer through weights $w_{f (n), H_{1} (n)}$ , and encodes the results of feature tests evaluated for each split node of the DT (numbered 0,1,4). The weights $w_{H_{1} (n), H_{2} (l)}$ between the two hidden layers encode the structure of the tree. In particular, the split nodes along the path $P (l)$ are connected to $H_{2} (l)$ . For example, leaf node 5 is connected to split node 0, but not split node 1. The second hidden layer encodes leaf membership for each leaf node (numbered 2,3,5,6). The final weights $w_{H_{2} (l), c}$ are fully connected and store the votes $y_{c}^{l}$ for each leaf $l$ and class $c$ . Gray: Input feature nodes. Blue: Bias nodes. Red: Prediction nodes, $p (c | x)$ . (c) NN corresponding to a RF with two DTs, each with the same architecture as in (a). Note that, while the two DTs have the same architecture, they use different input features at each split node, and do not share weights.


(a)	(b)	(c)

Hidden Layer 1. The first hidden layer, $H_{1}$

, is constructed with one neuron,

H_{1} (n)

, per split node in the corresponding DT. This neuron evaluates

x_{f (n)} \geq θ (n)

, and encodes the outcome in its activity,

a (H_{1} (n))

H_{1}

is connected to the input layer with the following weights and biases:

w_{f (n), H_{1} (n)} = s t r_{01}

and

b_{H_{1} (n)} = - s t r_{01} \cdot θ (n)

. The global constant

s t r_{01}

sets how rapidly the neuron activation changes as its input crosses its threshold. All other weights in this layer are zero.

As activation function in

H_{1}

a (\cdot) = tanh (\cdot)

is used, with a large value for

s t r_{01}

to approximate thresholded split decisions. During training,

s t r_{01}

can be reduced to avoid the problem of diminishing gradients in back-propagation; however, for now we assume

s t r_{01}

is a large positive constant. Thus, the pattern of activations encodes leaf node membership as follows:

x \in X_{l} ⟺ \forall n \in P (l) : {\begin{matrix} a (H_{1} (n)) = - 1 if X_{l} \subseteq X_{c l (n)} a (H_{1} (n)) = + 1 if X_{l} \subseteq X_{c r (n)} \end{matrix}

(3)

Hidden Layer 2. The role of neurons in the second hidden layer, $H_{2}$ , is to interpret the activation pattern a feature vector $x$ triggers in $H_{1}$ , and thus identify the unique $leaf (x)$ . Therefore, for every leaf $l$ in the DT, one neuron is created, denoted as $H_{2} (l)$ . Each such neuron is connected to all $H_{1} (n)$ with $n \in P (l)$ , but no others. Weights are set as follows: $w_{H_{1} (n), H_{2} (l)} = - s t r_{12}$ if $X_{l} \subseteq X_{c l (n)}$ and $w_{H_{1} (n), H_{2} (l)} = + s t r_{12}$ if $X_{l} \subseteq X_{c r (n)}$ . The sign of these weights matches the pattern of incoming activations iff $x \in X_{l}$ , thus making the activation of $H_{2} (l)$ maximal. To distinguish leaf membership, the biases in $H_{2}$ are set as $b_{H_{2} (l)} = - s t r_{12} \cdot (| P (l) | - 1)$ . Thus the input to node $H_{2} (l)$ is equal to $1$ if $x \in X_{l}$ , and less than or equal to $- 1$ otherwise. Using $tanh$ activation functions, linearly scaled to $[0, 1]$ range, and a large value for $s t r_{12}$ , the neurons approximately behave as binary switches that indicate leaf membership. I.e., $a (H_{2} (leaf (x))) = 1$ and all other neurons are silent.

Output Layer. The output layer of the NN has $C$ neurons, one for every class label. This layer is fully connected; however, there are no bias nodes introduced. The weights store scaled votes from the leaves of the corresponding DT: $w_{H_{2} (l), c} = s t r_{23} \cdot y_{c}^{l}$ . A softmax activation function is applied, to ensure a probabilistic interpretation of the output after training:

p (c | x) = \frac{e x p (s t r_{23} \cdot y_{c}^{leaf (x)})}{\sum_{c = 1}^{C} e x p (s t r_{23} \cdot y_{c}^{leaf (x)})}

(4)

Note that the softmax activation slightly perturbs the output distribution of the original RF (cf. Equation 2), making the mapping approximate. This can be tuned by the choice of $s t r_{23}$ , and in practice is a minor effect. Importantly, the softmax activation preserves the MAP solution.

From a Tree to a Forest. Let the number of DTs in a forest be denoted as $T$ . The prediction of a forest for feature vector $x$ to be of class $c$ is the normalised sum over the votes stored in the single leaf per tree $t$ , denoted ${leaf}_{t} (x)$ :

p (c | x) = \frac{\sum_{t = 1}^{T} y_{c}^{{leaf}_{t} (x)}}{\sum_{c = 1}^{C} \sum_{t = 1}^{T} y_{c}^{{leaf}_{t} (x)}}

(5)

Extending the DT-to-NN mapping described above to RFs is trivial: (i) replicate the basic NN design $T$ number of times, and (ii) fully connect $H_{2}$ to the output layer (see Figure 2(c)). This accomplishes summing over the leaf distributions from the different trees, before the softmax activation is applied.

3.2 Relationship Between RFs and CNNs

We now explain a new relationship which is crucial for our main contributions in Sections 3.3 and 3.4. The key concepts are summarized in Figure 3.

Figure 3: CNN architecture of a RF. (a) CNN architecture for dense semantic segmentation, corresponding to a RF with contextual features. The variables are: h - size of input convolution kernels; F - number of input convolution kernels; w - window size for offset features; d - number of feature maps in each layer; D - depth of corresponding decision tree; C - number of classes. (b) An example, where the RF is a single DT with depth D = 3, and 2 output classes. One pixel is classified in (b), corresponding to the region in (a) with similar color-coding. The input layer (red) extracts features with a fixed offset (shown by arrows) and filter type (index into filter stack, shown at bottom left of each node). Activation values are shown for nodes in $H_{1}$ , $H_{2}$ and the output layer. In this example, $leaf (x) = 5$ , highlighted by bold edges leading to the output layer. Bias nodes are not shown for simplicity.


(a)	(b)

One of the defining characteristics of CNNs is weight sharing across neurons corresponding to the same feature map. These neurons compute convolutions over a local window in their input, and their convolutional weights are constant across the entire feature map. Unsurprisingly, RFs work in the same way: A feature vector is pre-computed for each pixel in the image, and then fed through the same forest, or in the NN formulation given above, it traverses the identical NN.

A difference between RF and CNN is that in a RF, the first “convolutional layer” is pre-computed with a hand-selected filter bank, not learned as in a CNN. However, the subsequent operations of the RF can be broken down into two convolutions (corresponding to $H_{1}$ and $H_{2}$ ). The first of these two convolutions has depth equal to the number of filters in the filter bank, denoted F (typically $10 - 1000$ s), and is very sparse. E.g., axis-aligned decision stumps correspond to a convolution kernel with a single non-zero element. The second convolution ( $H_{2}$ ) is similarly very sparse, with the number of non-zero elements equal to the depth of the tree. Recall from Section 3.1, each neuron $H_{2} (l)$ in this layer combines the response of all split node neurons along path $P (l)$ . For instance, for a balanced tree of depth 10, $H_{1}$ creates $1023$ feature maps, where each neuron has a single input. $H_{2}$ creates $1024$ feature maps, but each neuron combines 10 features from the previous layer.

In many applications such as body-pose estimation

[23], medical image labeling [20], and scene labeling [32], contextual information is included in the form of contextual “offset features” that are selected from within a window defined by a maximum offset,

Δ_{m a x}

. In this case, neurons in

H_{1}

compute sparse convolutions with width and height of

2 * Δ_{m a x} + 1

, and depth F. Again, it is conventional to have only a single non-zero element in this convolution kernel; however, in the case of medical imaging it is also common to use e.g., average intensity over an offset window [20].

Altogether, a RF with contextual features can be viewed as a special case of a CNN, with sparse convolutional kernels and no max pooling layers. As we shall see in the next section, stacked RFs iterate this architecture using the previous RF predictions as input features, thereby generating a deep CNN with sparse convolutional kernels.

3.3 Mapping a RF Stack to a Deep CNN

In a stack of RFs, the modular architecture of a single RF is repeated. We map this architecture onto a deep CNN as follows: Each RF is mapped to a CNN, and then these CNNs are concatenated such that the layers corresponding to intermediate RF predictions become hidden layers, used as input to the next CNN in the sequence (see Figure 4). For a $K$ -level RF stack, this generates a deep CNN with $3 K - 1$ hidden layers. In the original Auto-context algorithm [32], each classifier can either select a feature from the output of the previous classifier, or from the set of input filter responses. Thus, we also introduce the input filter responses as bias nodes in hidden layers $H_{3 k}$ , $k = 1... K - 1$ . Note that both addition of trees to the RF and/or growing trees to a greater depth results in a CNN with 2 hidden layers, but with greater width. However, stacking RFs naturally increases the depth of the CNN architecture.

An interesteing question is what activation function to use on layers $H_{3 k}$ , which are no longer prediction layers. We explored the following options: identity, $tanh$ , class normalization (Equation 5), and softmax. Despite the fact that class normalization can in principle become undefined, due to the possibility of having negative weights, we found that it out-performed the other options. In particular, softmax was the most problematic, because it perturbs the prediction with regards to the original RF, and this error is compounded in a deep stack. This is consistent with class normalization performing the best, since it exactly matches the operation in the original RF stack. For the rest of the paper, we use class normalization activation functions on layers $H_{3 k}$

. We apply softmax activation at the final output layer to convert to a probability.

Figure 4: Mapping from a stacked RF to a deep CNN. (a) A stacked RF consisting of 2 shallow decision trees. The second RF takes as input the original stack of convolutional filter responses, and the output of the previous RF. (b) Corresponding CNN with 5 hidden layers. Same color coding and node labeling as in Figure 2. In this example, the second DT learned to use filter response $x_{2}$ , the RF output for class 1 at that pixel (*i.e.* $p_{1}$ ), and the RF output for class 2 at some different offset pixel, denoted ${¯ p}_{2}$ . Note that ${¯ p}_{2}$ is not a bias node; its value depends on weights in previous layers.

In stacked RFs used for semantic segmentation, individual pixels cannot be run through the entire stack independently, but rather the complete image must be run through one level at a time, such that all features are available for the next level. This is similarly true for our deep CNN.

3.4 Mapping the Deep CNN back to a RF Stack

We are interested in mapping our deep CNN architecture back to a stacked RF, with axis-aligned split functions, for fast evaluation at test time. Given a CNN constructed from a K-level RF stack as described above, the weights $w_{H_{3 k - 2, 3 k - 1}}$ , $k = 1... K$ manifest the correspondence of the CNN with the original tree structure. Thus, during training, keeping these weights and the corresponding biases, $b_{H_{3 k - 1}}$ fixed, allows the CNN to be trivially mapped back to the original RF stack. For a single level stack, the mapping is: (i) $θ (n) = - b_{H_{1} (n)} / w_{f (n), H_{1} (n)}$ , (ii) $y_{c}^{l} = w_{H_{2} (l), c}$ . We refer to this as “Map Back #1”. Finally, when evaluating this RF, a softmax activation function needs to be applied to the output distribution. For deeper stacks, the output of each RF must be post-processed with the corresponding activation function in the CNN, which in this paper is simple class normalization, but could be something different, such as softmax.

While the approach described above does map the CNN architecture back to the original RF stack, it may not make optimal use of the parameter refinement learned during back-propagation. Above, for a single level stack we assigned $y_{c}^{l} = w_{H_{2} (l), c}$ , which is the correct thing to do if only a single leaf neuron $H_{2} (l)$ fires in the network. However, after training by back-propagation, the activation pattern in $H_{2}$ may be distributed, with many neurons contributing to the prediction.

Here, we propose a strategy to capture the distributed activation of the CNN by updating the votes stored in the RF leaves. For feature vector $x$ and class $c$ , we would ideally like to store in ${leaf}_{t} (x)$ , the inner product of the activation pattern in $H_{2}$ with the out-going weights, $z_{x} (c) := \sum_{l \in N^{L e a f s}} a_{x} (H_{2} (l)) \cdot w_{H_{2} (l), c}$ .

This would elicit the identical output from the RF as from the CNN for input $x$ . However, the activation pattern will vary for different training samples that end up in the same leaf, so this mapping cannot be satisfied simultaneously for the whole training set. In other words, DTs store distributions in their leaves $l$ that represent constant functions on the respective $X_{l}$ , while the re-trained CNN allows for non-constant functions on $X_{l}$ (see Figure 5). As a compromise, we seek new vote distributions ${^y}_{c}^{l}$ , for each $c, l,$ to minimise the following error, averaged over the finite set of training samples, $X^{t r a i n} \subset X$ .

(6)

Equation 6 can be solved analytically, yielding the following result:

^ylc=1|{x∈Xtrain: leaf(x)=l}|∑x∈Xtrain: leaf(x)=lzx(c)

(7)

This is a simple average of $z_{x} (c)$ over all samples that end up in the same leaf of the corresponding DT. We refer to this as “Map Back #2”. In the trivial case where, for every sample, only one neuron fires in $H_{2}$ , this is equivalent to “Map Back #1”.

Figure 5: Mapping CNN back to a RF. (a) Three samples (blue, magenta, green) falling into the leaf of a DT, corresponding to a subset $X_{l}$ of feature space, have the same posterior distributions; however, in a CNN their posteriors can be different. (b) Corresponding activation pattern $a (H_{2} (l))$ for the three samples shown in (a) at $H_{2}$ of RF initialized NN. The output layer receives the inner product of the activation pattern with weights $w_{H_{2} (l), c}$ (only weights to class $1$ shown with black line for simplicity). (c) Activation pattern in corresponding RF. Note, the inner product reduces to the value $y_{1}^{l}$ for class $1$ . In Equation 7, we compute the optimal value of $y_{1}^{l}$ to mimize the difference between the RF and the CNN.


(a)	(b)	(c)

To implement this algorithm in a stack, we must take one additional precaution. Since updating the votes as described in Equation 7 does not capture the output of the re-trained CNN exactly, we update the votes sequentially, from the first to the last level of the corresponding stack. E.g., for a 2 level stack, after updating the votes in the first RF using Equation 7, we pass the training data through and determine the new value of $leaf (x)$ in the second RF for each training sample, and use this to update the votes in the second RF. See Algorithm 1 for details.

1. Push all training data through CNN

2. Store

a_{x} (H_{3 k - 1} (l))

, for

k = 1... K

for

i = 1 : K

Push all training data through stacked RF to level

i

Store

{leaf}_{t} (x)

, at level i

Update votes in

i^{t h}

RF to

{^y}_{c}^{l}

, according to Equation 7

end for

Algorithm 1 Algorithm for mapping deep CNN back to K-level stacked RF. The following algorithm was used to map the parameters from a trained CNN back to the original stacked RF architecture. We applied this algorithm to the zebrafish data set (Figure 1(panel 3) and Figure 8(f)).

4 Results

4.1 Kinect Body Part Classification

Experimental Setup. We applied our method to human body part classification from Kinect depth images, a domain where Random Forests have been highly successful [28]. We use the recently provided data set in [8], since there is no publicly available data set from the original paper [28]. It contains 2000 training images, and 500 testing images, each 320x240 pixels, containing 19 foreground classes and 1 background class (see Figure 6(a,b) for an example). We evaluate the pixel accuracy, averaged over all foreground classes, as was done by [8]. Note that background is trivially classified.

Training Parameters. We first trained a two-level stacked RF, and then mapped the RF stack to a deep CNN with 5 hidden layers, as described in Section 3

. We trained the CNN using back-propagation and stochastic gradient descent (SGD) with momentum. SGD training is applied by passing images through the network one at a time, and computing the gradient averaged over all pixels (

i.e., batch size = 1 image). Thus, we do “whole-image-at-a-time” training, as in [19]. We trained for 8000 iterations, which takes approximately

10

hours in our CPU-based Matlab implementation. For a detailed list of the parameters, see Section 6.1.1.

Results. With our initial two-level stacked RF, we achieved a pixel accuracy of $0.82$ , comparable to the original result of 0.79 [8] (See Figure 6(c)). After mapping to a deep CNN and re-training, we achieved a pixel accuracy of $0.91$ , corresponding to an $11 %$ relative improvement over the RF stack (see Figure 6(d)). This final result is comparable to the state-of-the-art result on this data set which aims to compress RFs by learning a better combination of their constituent trees [23]. They achieve a class-balanced pixel accuracy of $0.92$ over all classes, including the background class, for a model size of 6.8MB. Our model is smaller, at 3.3MB, due to our use of fewer and shallower trees. Due to the different error metric, and their evaluation on a selected subset of pixels, the results are not directly comparable; however, they appear to be very similar.


(a)	(b)	(c)	(d)	(e)

Figure 6: Example result of Kinect body part classification. (a) Depth image input to pixel classifier. (b) Ground truth labeling. (c) Result of stacked RF. (d) Result of RF-initialized CNN, after re-training. The accuracy for this test image increases from 0.88 to 0.94 on foreground classes with the CNN. (e) Crop of hands for GT, RF and CNN, from top to bottom. Note the improvement of small parts, e.g. hands.

Insights.

The architecture of the deep CNN preserves the intermediate prediction layers of the RF stack, which generates one image for each class at the same resolution as the input image. This enables us to gain insights on internal CNN layers. However, due to back-propagation training, these images no longer represent probability distributions. In particular, the pixel values can now be negative. We visualized the internal layers to better understand how they changed during additional training in the CNN (Figure

7). Interestingly, we noticed that compared to the stacked RF, the internal activation layers in the CNN were less thresholded, and fired on adjacent body parts. A common strategy in stacked classification is to introduce smoothing between the layers of the stack (see e.g. [15, 13, 24]), and it appears that a similar strategy is naturally learned by the deep CNN.

Figure 7: Visualization of internal activation layers in body part classifiers. We visualize the probability map output by the first level of a two-level RF stack (Level 1 Output), and the corresponding activation map from hidden layer 3 of the CNN ( $H_{3}$ ) for two exemplary classes. Notice that the activation maps in the CNN are smoothed over adjacent body regions, whereas in the stacked RF they are sharply focused. Best viewed in colour.

4.2 Zebrafish Somite Classification

Experimental Setup. We next applied our method to semantic segmentation of 21 somites and 1 background class in a data set of 32 images (800x950 pixels) of developing zebrafish¹¹1Somites are the metameric units that give rise to muscle and bone, including vertebrae.^,²²2This data set will be made publicly available upon acceptance of the manuscript. Experts in biology manually created ground truth segmentations of these images. This data set poses multiple challenges for automated segmentation, due to the similar appearance of neighboring segments and the limited training data. The data set was split into 16 images for training and 16 images for test. Two additional training images were generated from each original training image by random rotation of the originals. We evaluated the resulting segmentation by means of the class-balanced Dice score.

Training Parameters. We first trained a three-level stacked RF, and then mapped the RF stack to a deep CNN with 8 hidden layers. The CNN was initialized and trained exactly as for the Kinect example; however, with different parameters (see Section 6.1.2).

Results. Segmentation of the test data by means of the resulting three-level stacked RF achieved an average Dice score of 0.60 (see Figure 8(c) and Table 1(RF)). The RF-initialized CNN achieved a Dice score of 0.66 after re-training, corresponding to a $10 %$ relative improvement (see Figure 8(d) and Table 1(CNN)).

Next, we mapped the CNN back to the initial stacked RF architecture, albeit with updated parameters, for fast test-time evaluation. We first employed the trivial approach of mapping weights directly onto votes, similar to what was done in the RF to NN mapping; however, this reduced the Dice score to $0.59$ (see Figure 8(e) and Table 1(MB1)), worse than the performance of the initial RF. Next we applied Algorithm 1, which produces a result that is visually superior to the trivial mapping, and yields a final Dice score of $0.63$ (see Figure 8(f) and Table 1(MB2)). Thus, we achieve a $5 %$ relative improvement of our RF stack, which retains its exact tree structure, by mapping to a deep CNN, training all weights by back-propagation, and mapping back to the original RF stack with updated threshold and leaf distributions.

Figure 8: Comparison of different methods for zebrafish somite labeling. (a) Raw image of zebrafish. Yellow box denotes crop for b,c,d,f. (b) Ground truth labeling. (c) Prediction of stacked RF. (d) Prediction of corresponding deep CNN, after parameter refinement by back-propagation. (e) Prediction of “Map Back #1” stacked RF. (f) Prediction of “Map Back #2” stacked RF. See Section 3.4 for details of map back algorithms.


(a)	(b)	(c)	(d)	(e)	(f)

Method	RF	FCN	CNN	MB1	MB2
Dice Score	0.60	0.18	0.66	0.59	0.63

Table 1: Comparison of dense semantic labeling of zebrafish somites by different methods. Dice score is reported for the output of the initial stacked RF (RF), Fully Convolutional Network (FCN)[19], RF-initialized and re-trained CNN (CNN), and after mapping the CNN back to a stacked RF using Map Back #1 and #2 (MB1 and MB2, respectively). See Section 3.4 for details of mapping back. Higher Dice score corresponds to a more accurate segmentation.

Above we described a method for training a deep CNN on relatively little training data, using a novel initialization from a stacked RF. As a comparison, we considered the task of training the same CNN architecture from a random initialization, using a similar SGD training routine (see Section 6.1.2 for parameters). We first attempted to train the network maintaining the sparsity of the weight layers. However, the energy quickly plateaued, and yielded a final Dice score of only $0.04$ . We then fully connected the layers corresponding to the tree connectivity, (i.e. $w_{H_{1} (n), H_{2} (l)}$ , $w_{H_{4} (n), H_{5} (l)}$ , $w_{H_{7} (n), H_{8} (l)}$ ) and retrained with the same hyper-parameters. This network performed considerably better, reaching a final Dice score of $0.15$ .

We also compared our method with the Fully Convolutional Network (FCN), a state-of-the-art method for semantic segmentation using CNNs [19]

. This network was downloaded from Caffe’s Model Zoo

³³3https://github.com/BVLC/caffe/wiki/Model-Zoo#fcn, and initialized with weights fine-tuned from the ILSVRC-trained VGG-16 model. Fine-tuning takes approximately

1

day on a single Nvidia K-40 GPU (see Section 6.1.2 for details). We observed that the FCN network failed to train successfully, achieving a Dice score of only

0.18

, likely because of the limited size of the training data set (see Figure 9).

Figure 9: Result of zebrafish somite labeling using FCN [19]. (a) Ground truth labeling. (b) Prediction of FCN. The average Dice score for FCN was 0.18 on the test set, compared to 0.66 for our RF-initialized deep CNN. This representative image shows that FCN manages to learn the approximate locations of each class from the strong contextual information in the image; however, it fails to return an accurate segmentation.


(a)	(b)

Insights. In Figure 10 we discuss insights on the internal activation layers of this network.

Figure 10: Visualization of internal activation layers in somite classifiers. We visualize the probability maps output by the first two levels of a three-level RF stack (Level 1,2 Output), and the corresponding activation maps from hidden layers 3 and 6 of the CNN ( $H_{3}$ , $H_{6}$ ) for somite $# 7$ . Notice that the activation from the CNN appears smoothed along the direction of the foreground classes compared to the noisier output of the stacked RF. Best viewed in colour.

5 Conclusions and Future Work

We have exploited a new mapping between stacked RFs and Deep CNNs, and demonstrated the practical benefits of this mapping for semantic segmentation. This is particularly important when dealing with limited amount of training data. In contrast to common CNN architectures, our specific architecture produces internal activation images, one for each class, which are of the same dimension as the input image. This enables us to gain insights on the semantic behaviour of the internal layers.

There are many exciting avenues for future research. In the short term, we plan to refine the input convolution filters, which are currently fixed, during back-propagation. Another refinement is to incorporate drop-out regularization during training, which should lead to better generalization performance as has been shown for traditional CNN architectures. Also, the approximate mapping from a CNN architecture back to stacked RFs, and related test-time efficient architectures, may be further improved. In the midterm we are excited about extending our architecture and also merging it with existing CNN architectures. Since our internal activation images are directly interpretable, it is straight forward to incorporate differentiable model layers. It will be interesting to see how our specialized CNN behaves as part of a larger CNN network, for instance by placing it directly after the feature extraction layers of a traditional CNN.

6 Supplemental Materials

In Section 6.1.1, we describe the training parameters used to train the stacked RF and deep CNN for the Kinect example. In Section 6.1.2, we describe the training parameters used to train the stacked RF and deep CNN for the zebrafish example. We also describe the parameters used for training the equivalent deep CNN with random weight initialization.

6.1 Training Parameters

6.1.1 Kinect

Stacked RF. We trained a two-level stacked RF, with the following forest parameters at every level: 10 trees, maximum depth 12, stop node splitting if less than 25 samples. We selected 20 samples per class per image for training, and used the standard scale invariant offset features from [28]

, with standard deviation,

σ

= 50 in each dimension. Each split node selected the best from a random sample of 100 such features.

CNN. We mapped the RF stack to a deep CNN with 5 hidden layers, as described in Section 3.3. For efficient training, the initialization parameters $s t r_{01} - s t r_{89}$ were reduced such that the network could transmit a strong gradient via back-propagation. However, softening these parameters moves the deep CNN further from its initialization by the equivalent stacked RF. We evaluated a range of initialization parameters and found $s t r_{01} = s t r_{34} = s t r_{67} = 100$ , $s t r_{12} = s t r_{45} = s t r_{78} = 1$ , $s t r_{23} = s t r_{56} = s t r_{89} = 0.1$ to be a good compromise.

We trained the CNN using back-propagation and stochastic gradient descent (SGD), with a cross-entropy loss function. During back-propagation, we maintained the sparse connectivity from RF initialization, allowing only the weights on pre-existing edges to change, corresponding to the sparse training scheme from [33].

Since the network is designed for whole-image inputs, we first cropped the training images around the region of foreground pixels, and then down-sampled them by 25x. Learning rate, $r$ , was set such that for the $i^{t h}$ iteration of SGD, $r (i) = a (1 + i / b)^{- 1}$ with hyper-parameters $a = 0.01$ and $b = 400$ iterations. Momentum, $μ$ , was set according to the following schedule: $μ = m i n {μ_{m a x}, 1 - 3 / (i + 5)}$ , where $μ_{m a x} = 0.95$ [30].

6.1.2 Zebrafish

Stacked RF.

We trained a three-level RF stack, with the following forest parameters at every level: 16 trees, maximum depth 12, stop node splitting if less than 25 samples. Features were extracted from the images using a standard filter bank, and then normalized to zero mean, unit variance. The number of random features tested in each node was set to the square root of the total number of input features. For each randomly selected feature, 10 additional contextual features were also considered, with X and Y offsets within a 129x129 pixel window. Training samples were generated by sub-sampling the training images 3x in each dimension and then randomly selecting

25 %

of these samples for training.

CNN. We mapped the RF stack to a deep CNN with 8 hidden layers. The CNN was initialized and trained exactly as for the Kinect example, with the following exeptions: (i) We used a class-balanced cross-entropy loss function, (ii) Training samples were generated by sub-sampling the training images 9x in each dimension. (iii) Learning rate parameters were as follows: $a = 0.01$ and $b = 96$ iterations. (iv) Momentum was initialized to $μ = 0.4$ , and increased to $0.7$ after $96$ iterations. We observed convergence after only 1-2 passes through the training data, similar to what was reported by [12].

CNN from Random Initialization.

As discussed in Section 4.2 of the paper, for comparison to the RF-initialized weights described above, we also trained CNNs with the same architecture, but with random weight initialization. Weights were initialized according to a Gaussian distribution with zero mean and standard deviation,

σ = 0.01

. We applied a similar SGD training routine, and re-tuned the hyper-parameters as follows:

a = 3 * 10^{- 5}

b = 96

iterations, momentum was initialized to 0.4 and increased to 0.99 after 96 iterations. Larger step-sizes failed to train. Networks were trained for 2500 iterations.

Fully Convolutional Network. As discussed in Section 4.2 of the paper, we also compared our method with the Fully Convolutional Network (FCN) [19]. This network was downloaded from Caffe’s Model Zoo⁴⁴4https://github.com/BVLC/caffe/wiki/Model-Zoo#fcn, and initialized with weights fine-tuned from the ILSVRC-trained VGG-16 model. We trained all layers of the network using SGD with a learning rate of $10^{- 9}$ , momentum of $0.99$ and weight decay of $0.0005$ . See Figure 9(b) for an example of the resulting segmentation.

References

[1] Amr Ahmed, Kai Yu, Wei Xu, Yihong Gong, and Eric Xing. Training hierarchical feed-forward visual recognition models using transfer learning from pseudo-tasks. In ECCV, 2008.
[2] Yoshua Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the Trade, pages 437–478, 2012.
[3] Leon Bottou. Stochastic gradient descent tricks. In Neural Networks: Tricks of the Trade, pages 421–436, 2012.
[4] Leo Breiman. Random forests. Machine Learning, 45(1):5–32, 2001.
[5] Samuel Rota Bulò and Peter Kontschieder. Neural decision forests for semantic image labelling. In CVPR, 2014.
[6] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. In ICLR, 2015.
[7] Dan Ciresan, Alessandro Giusti, Luca M Gambardella, and Jürgen Schmidhuber. Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images. In NIPS, pages 2852–2860, 2012.
[8] Misha Denil, David Matheson, and Nando de Freitas. Consistency of online random forests. pages 1256–1264, 2013.
[9] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In NIPS, 2014.
[10] Scott Fahlman and Christian Lebiere. The cascade-correlation learning architecture. In NIPS, pages 524–532, 1989.
[11] Clement Farabet, Camille Couprie, Laurent Najman, and Yann LeCun. Learning hierarchical features for scene labeling. PAMI, 35(8):1915–1929, 2013.
[12] Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
[13] Varun Jampani, Raghudeep Gadde, and Peter Gehler. Efficient facade segmentation using auto-context. In WACV, 2015.
[14] Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulo. Deep neural decision forests. In ICCV, 2015.
[15] Peter Kontschieder, Pushmeet Kohli, Jamie Shotton, and Antonio Criminisi. Geof: Geodesic forests for learning coupled predictors. In CVPR, pages 65–72, 2013.
[16] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
[17] Chen Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. In AISTATS, 2015.
[18] Régis Lengellé and Thierry Denoeux. Training mlps layer by layer using an objective function for internal representations. Neural Networks, 9(1):83–97, 1996.
[19] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
[20] A Montillo, J Tu, J Shotton, J Winn, JE Iglesias, DN Metaxas, and A Criminisi. Entanglement and differentiable information gain maximization. In Decision Forests for Computer Vision and Medical Image Analysis, 2013.
[21] Feng Ning, Damien Delhomme, Yann LeCun, Fabio Piano, Leon Bottou, and Paolo Emilio Barbano. Toward automatic phenotyping of developing embryos from videos. IEEE Trans. Image Processing, 14(9):1360–1371, 2005.
[22] Marc Aurelio Ranzato, Fu Jie Huang, Y-Lan Boureau, and Yann LeCun. Unsupervised learning of invariant feature hierarchies with applications to object recognition. In CVPR, 2007.
[23] Shaoqing Ren, Xudong Cao, Yichen Wei, and Jian Sun. Global refinement of random forest. In CVPR, 2015.
[24] David Richmond, Dagmar Kainmueller, Ben Glocker, Carsten Rother, and Gene Myers. Uncertainty-driven forest predictors for vertebra localization and segmentation. In MICCAI, pages 653–660, 2015.
[25] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234–241, 2015.
[26] Pierre Sermanet, Koray Kavukcuoglu, Sumith Chintala, and Yann LeCun. Pedestrian detection with unsupervised multi-stage feature learning. In CVPR, 2013.
[27] Ishwar Sethi. Entropy nets: from decision trees to neural networks. Proceedings of the IEEE, 78(10):1605–1613, 1990.
[28] Jamie Shotton, Andrew W. Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, and Andrew Blake. Real-time human pose recognition in parts from single depth images. In CVPR, 2011.
[29] Jamie Shotton, Matthew Johnson, and Roberto Cipolla. Semantic texton forests for image categorization and segmentation. In CVPR, 2008.
[30] Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton.
On the importance of initialization and momentum in deep learning.
In ICML, 2013.
[31] Alberto Suárez and James Lutsko. Globally optimal fuzzy decision trees for classification and regression. PAMI, 21(12):1297–1311, 1999.
[32] Zhuowen Tu and Xiang Bai. Auto-context and its application to high-level vision tasks and 3d brain image segmentation. PAMI, 32(10):1744–1757, 2010.
[33] Johannes Welbl. Casting random forests as artificial neural networks (and profiting from it). In GCPR, 2014.
[34] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip Torr. Conditional random fields as recurrent neural networks. In ICCV, 2015.

Relating Cascaded Random Forests to Deep Convolutional Neural Networks for Semantic Segmentation

Authors

Learning Semantic Segmentation with Diverse Supervision

Semantic Segmentation with Labeling Uncertainty and Class Imbalance

Understanding Convolution for Semantic Segmentation

Egocentric Human Segmentation for Mixed Reality

A Survey of Semantic Segmentation

Real Time Egocentric Object Segmentation: THU-READ Labeling and Benchmarking Results

Solar Potential Assessment using Multi-Class Buildings Segmentation from Aerial Images

Code Repositories

awesome-random-forest

1 Introduction

2 Related Work

3 Method