Manifold Forests: Closing the Gap on Neural Networks

1 Introduction

Decision forests, including random forests and gradient boosting trees, have solidified themselves in the past couple decades as a powerful ensemble learning method in supervised settings [JMLR:v15:delgado14a, Caruana:2006:ECS:1143844.1143865], including both classification and regression [hastie01statisticallearning]

. In classification, each forest is a collection of decision trees whose individual classifications of a data point are aggregated together using majority vote. One of the strengths of this approach is that each decision tree need only perform better than chance for the forest to be a strong learner, given a few assumptions

[Schapire:1990:SWL:83637.83645, Biau:2008:CRF:1390681.1442799]. Additionally, decision trees are relatively interpretable because they can provide an understanding of which features are most important for correct classification [Breiman2001]

. Breiman originally proposed decision trees that partition the data set using hyperplanes aligned to feature axes

[Breiman2001]

. Yet, this limits the flexibility of the forest and requires deep trees to classify some data sets, leading to overfitting. He also suggested that algorithms which partition based on sparse linear combinations of the coordinate axes can improve performance

[Breiman2001]. More recently, Sparse Projection Oblique Randomer Forest (), partitions a random projection of the data and has shown impressive improvement over other methods [SPORF].

Yet random forests and other machine learning algorithms frequently operate in a tabular setting, viewing an observation $→x=(x1,…,xp)\T∈Rp$

as an unstructured feature vector. In doing so, they neglect the indices in settings where the indices encode additional information. For structured data, e.g. images or time series, traditional decision forests are not able to incorporate known continuity between features to learn new features. For decision forests to utilize known local structure in data, new features encoding this information must be manually constructed. Prior research has extended random forests to a variety of computer vision tasks

[rf_keypoint_recog, rf_hough_detection, rf_image_classification, kinect_rf] and augmented random forests with structured pixel label information [rf_structured]. Yet these methods either generate features a priori from individual pixels, and thus do not take advantage of the local topology, or lack the flexibility to learn relevant patches. Decision forests have been used to learn distance metrics on unknown manifolds [Criminisi:2012:DFU:2185837.2185838], but such manifold forest algorithms are unsupervised and aim to learn a low dimensional representation of the data.

Inspired by , we propose a projection distribution that takes into account continuity between neighboring features while incorporating enough randomness to learn relevant projections. At each node in the decision tree, sets of random spatially contiguous features are randomly selected using knowledge of the underlying manifold. Summing the intensities of the sampled features yields a set of projections which can then be evaluated to partition the observations. We describe this proposed classification algorithm, Manifold Forests () in detail and show its effectiveness in three simulation settings as compared to common classification algorithms. Furthermore, the optimized and parallelizable open source implementation of in R and Python is available ¹¹1https://neurodata.io/sporf/. This addition makes for an effective and flexible learner across a wide range of manifold structures.

2 Background and Related Work

2.1 Classification

In the two-class classification setting, there is a data set $D = {(x_{i}, y_{i})}_{i = 1}^{n}$ of n pairs $(x_{i}, y_{i})$ drawn from an unknown distribution $F_{X Y}$ where $x_{i} \in X \subset R^{p}$ and $y_{i} \in Y = {0, 1}$ . Our goal is to train a classifier $h (x; D_{n}) : X \times (X \times Y)^{n} \to Y$ based on our observations that generalizes to correctly predict the class of an observed $x$

. The performance of this classifier is evaluated via the 0-1 Loss function

L (h (x), y) = I [h (x) \neq y]

to find the optimal classifier

h∗=\argminhE[L(h(x),y)]

, which minimizes the probability of an incorrect classification.

2.2 Random Forests

Originally popularized by Breiman, the random forest (RF) classifier is empirically very effective [JMLR:v15:delgado14a] while maintaining strong theoretical guarantees [Breiman2001]. A random forest is an ensemble of decision trees whose individual classifications of a data point are aggregated together using majority vote. Each decision tree consists of split nodes and leaf nodes. A split node is associated with a subset of the data $S = {(x_{i}, y_{i})} \subseteq D$ and splits into two child nodes, each associated with a binary partition of $S$ . Let $e_{j} \in R^{p}$ denote a unit vector in the standard basis (that is, a vector with a single one and the rest of the entries are zero) and $τ$ a threshold value. Then $S$ is partitioned into two subsets given the pair $θ_{j} = {e_{j}, τ}$ .

	$S_{θ}^{L}$	$={(xi,yi)\|ej\Txi<τ}$
	$S_{θ}^{R}$	$={(xi,yi)\|ej\Txi≥τ}$

To choose the partition, the optimal $θ^{*} = (e_{j}^{*}, τ^{*})$ pair is selected via a greedy search from among a set of $d$ randomly selected standard basis vectors $e_{j}$ . The selected partition is that which maximizes some measure of information gain. A typical measure is a decrease in impurity, calculated by the Gini impurity score $I (S)$ , of the resulting partitions [hastie01statisticallearning]. Let $^pk=1|S|∑yi∈S\II[yi=k]$ be the fraction of elements of class $k$ in partition $S$ , then the optimal split is found as

	$I (S)$	$= K \sum k = 1 {^p}_{k} (1 - {^p}_{k})$
	$θ^{*}$	$=\argmaxθ\|S\|I(S)−\|SLθ\|I(SLθ)−\|SRθ\|I(SRθ).$

A leaf node is created once the partition reaches a stopping criterion, typically either falling below an impurity score threshold or a minimum number of observations [hastie01statisticallearning]. The leaf nodes of the tree form a disjoint partition of the feature space in which each partition of observations $S_{k}$ is assigned a class label $c_{k}^{*}$ corresponding to the class majority.

c∗k=\argmaxck∑yi∈SI(yi=ck).

A decision tree classifies a new observation by assigning it the class of the partition into which the observation falls. The forest averages the classifications over all decision trees to make the final classification [hastie01statisticallearning]. For good performance of the ensemble and strong theoretical guarantees, the individual decision trees must be relatively uncorrelated from one another. Breiman’s random forest algorithm does this in two ways:

At every node in the decision tree, the optimal split is determined over a random subset $d$ of the total collection of features $p$ .
Each tree is trained on a randomly bootstrapped sample of data points $D^{'} \subset D$ from the full training data set.

Applying these techniques means that random forests do not overfit and lowers the upper bound of the generalization error [Breiman2001].

2.3 Sparse Projection Oblique Randomer Forests

is a recent modification to random forest that has shown improvement over other versions [SPORF, tomita2]. Recall that RF split nodes partition data along the coordinate axes by comparing the projection $ej\Txi$ of observation $x_{i}$ on standard basis $e_{j}$ to a threshold value $τ$ . generalizes the set of possible projections, allowing for the data to be partitioned along axes specified by any sparse vector $a$ .

	$S_{θ}^{L}$	$={(xi,yi)\|a\Txi<τ}$
	$S_{θ}^{R}$	$={(xi,yi)\|a\Txi≥τ}$

Rather than partitioning the data solely along the coordinate axes (i.e. the standard basis), creates partitions along axes specified by sparse vectors. In other words, let the dictionary $\mcA$ be the set of atoms ${a}$ , each atom a $p$ -dimensional vector defining a possible projection $a\Txi$ . In axis-aligned forests, $\mcA$ is the set of standard basis vectors ${e_{j}}$ . In , the dictionary $\mcD$ can be much larger, because it includes, for example, all 2-sparse vectors. At each split node, samples $d$ atoms from $\mcD$ according to a specified distribution. By default, each of the $d$

atoms are randomly generated with a sparsity level drawn from a Poisson distribution with a specified rate

λ

. Then, each of the non-zero elements are uniformly randomly assigned either

+ 1

- 1

. Note that the size of the dictionary for is

3^{p}

(because each of the

p

elements could be

- 1

0

, or

+ 1

), although the atoms are sampled from a distribution heavily skewed towards sparsity.

3 Methods

3.1 Random Projection Forests on Manifolds

In the structured setting, the dictionary of projection vectors $\mcA={a}$ is modified to take advantage of the underlying manifold on which the data lies. We term this method the Manifold Forest ().

Each atom $a$ projects an observation to a real number and is designed with respect to prior knowledge of the data manifold. Nonzero elements of $a$ effectively select and weight features. Since the feature space is structured, each element of $a$ maps to a location on the underlying manifold. Thus, patterns of contiguous points on the manifold define the atoms of $\mcA$ ; the distribution of those patterns yields a distribution over the atoms. At each node in the decision tree, samples $d$ atoms, yielding $d$ new features per observation. proceeds just like by optimizing the best split according to the Gini index. Algorithm pseudocode, essentially equivalent to that of , can be found in the Appendix.

In the case of two-dimensional arrays, such as images, an observation $x_{i} \in R^{p}$ is a vectorized representation of a data-matrix $X_{i} \in R^{W \times H}$ . To capture the relevance of neighboring pixels, creates projections by summing the intensities of pixels in rectangular patches. Thus the atoms of $\mcA$ are the vectorized representations of these rectangular patches.

A rectangular patch is fully parameterized by the location of its upper-left corner $(u, v)$ , its height $h$ , and width $w$

. To generate a patch, first the index of the upper left corner is uniformly sampled. Then its height and width are independently sampled from separate uniform distributions. hyperparameters determine the minimum and maximum heights heights

{h_{m i n}

h_{m a x}}

and widths

{w_{m i n}, w_{m a x}}

, respectively, to sample from. Let

u n i f {α, β}

denote the discrete uniform distribution. An atom

a

is sampled as follows. Note that the patch cannot exceed the data-matrix boundaries.

	$u \sim u n i f {0, W - w_{m i n}}$	$v \sim u n i f {0, H - h_{m i n}}$
	$w \sim u n i f {w_{m i n}, m i n (w_{m a x}, W - u)}$	$h \sim u n i f {h_{m i n}, m i n (h_{m a x}, H - v)}$

The vectorized atom $a$ yields a projection of the data $a\Txi$ , effectively selecting and summing pixel intensities in the sampled rectangular patch.

a\T=(0,…,0W×v,0,…,0u,(h−1)×W1,…,1w,0,…,0W−w,…,1,…,1w,0,…,0W−w,1,…,1w,0,…,0W−w−u,0,…,0W×(H−h−v))

By constructing features in this way, learns low-level features in the structured data, such as edges or corners in images. The forest can therefore learn the features that best distinguish a class. The structure of these atoms is flexible and task dependent. In the case of data lying on a cyclic manifold, the atoms can wrap-around borders to capture the added continuity. Atoms can also be used in one-dimensional arrays, such as univariate time-series data, in which case $h_{m i n} = h_{m a x} = 1$

3.2 Feature Importance

One of the benefits to decision trees is that their results are fairly interpretable in that they allow for estimation of the relative importance of each feature. Many approaches have been suggested

[Breiman2001, Lundberg2017AUA] and here a projection forest specific metric is used in which the number of times a given feature was used in projections across the ensemble of decision trees is counted. A decision tree

T

is composed of many nodes

k

, each one associated with an atom

a_{k}^{*}

and threshold that partition the feature space according to the projection

a_{k}^{*} \cdot x_{i}

. Thus, the indices corresponding to nonzero elements of

a_{k}^{*}

indicate important features used in the projection. For each feature

j

, the number of times

π_{j}

it is used in a projection, across all split nodes and decision trees, is counted.

π_{j} = \sum T \sum k \in T I (a_{k j}^{*} \neq 0)

These normalized counts represent the relative importance of each feature in making a correct classification. Such a method applies to both and , although different results between them would be expected due to different projection distributions yielding different hyperplanes.

4 Simulation Results

To test , we evaluate its performance in three simulation settings as compared to logistic regression (Log. Reg), linear support vector machine (Lin. SVM), support vector machine with a radial basis function kernel (SVM), k-nearest neighbors (kNN), random forest (RF), Multi-layer Perceptron (MLP), and (SPORF). For each experiment, we used our open source implementation of and that of . All decision forest algorithms used 100 decision trees on the simulations. Each of the other classifiers were run from the Scikit-learn Python package

[scikit-learn]

with default parameters. Additionally, we tested against a Convolutional Neural Network (CNN) built using PyTorch

[paszke2017automatic]

with two convolution layers, ReLU activations, and maxpooling, followed by dropout and two densely connected layers. The CNN results were averaged over 5 runs for the simulations and training was stopped early if the loss plateaued.

4.1 Simulation Settings

Experiment (A) is a non-Euclidean example inspired by Younes2018DiffeomorphicL. Each observation is a discretization of a circle into 100 features with two non-adjacent segments of 1’s in two differing patterns: class 1 features two segments of length five while class 2 features one segment of length four and one of length six. chose one-dimensional rectangles in this setting as the observations were one-dimensional in nature. These projection patches had a width between one and fifteen pixels and each split node of and considered 40 random projections. Figure 1(A) shows examples from the two classes and classification results across various sample sizes.

In experiment (B) consists of a simple $28 \times 28$ binary image classification problem. Images in class 0 contain randomly sized and spaced horizontal bars while those in class 1 contain randomly sized and spaced vertical bars. For each sampled image, $k \sim P o i s s o n (λ = 10)$ bars were distributed among the rows or columns, depending on the class. The distributions of the two classes are identical if a 90 degree rotation is applied to one of the classes. Projection patches were between one and four pixels in both width and height and each split node of and considered 28 random projections. Figure 1(B) shows examples from the two classes and classification results across various sample sizes.

Experiment (C) is a signal classification problem. One class consists of 100 values of Gaussian noise while the second class has an added exponentially decaying unit step beginning at time 20.

	$x_{t}^{(0)}$	$= ϵ_{t}$
	$x_{t}^{(1)}$	$= u (t - 20) e^{(t - 20)} + ϵ_{t}$
	$ϵ_{t}$	$\sim N (0, 1)$

Projection patches were 1D with a width between one and five timesteps. Each split node of and considered the default number of random projections, the square root of the number of features. Figure 1(C) shows examples from the two classes and classification results across various sample sizes.

4.2 Classification Accuracy

Figure 1: outperforms other algorithms in three two-class classification settings. Upper row shows examples of simulated data from each setting and class. Lower row shows misclassification rate in each setting, tested on 10,000 test samples. (A) Two segments in a discretized circle. Segment lengths vary by class. (B) Image setting with uniformly distributed horizontal or vertical bars. (C)White noise (class 0) vs. exponentially decaying unit impulse plus white noise (class 1).

In all three simulation settings, outperforms all other classifiers, doing especially better at low sample sizes, except the CNN for which there is no clear winner. The performance of is particularly good in the discretized circle simulation for which most other classifiers perform at chance levels. also performs well in the signal classification problem although all the classifiers are close in performance. This may be because the exponential signal is prevalent throughout most of the time-steps and so perfect continuity is less relevant.

Figure 2: Algorithm train times (above) and test times (below). runtime is not particularly costly and well below CNN runtime in most examples.

4.3 Run Time

All experiments were run on a single core CPU. has train and test times on par with those of and and so is not particularly more computationally intensive to run. The CNN, however, took noticeably longer to run—especially in terms of training, but also in testing—in two of the three simulations. Thus its strong performance in those settings comes at an added computational cost, a typical issue for deep learning methods

[dl_cost].

5 Real Data Results

5.1 Classification Accuracy

Figure 3: improves classification accuracy over all other non-CNN algorithms for all sample sizes, especially in small sample sizes.

’s performance was evaluated on the MNIST dataset, a collection of handwritten digits stored in 28 by 28 square images [mnist], and compared to the algorithms used in the simulations. 10,000 images were held out for testing and a subset of the remaining images were used for training. The results are displayed in Figure 3. All three forest algorithms were composed of 500 decision trees and was restricted to use patches up to only 3 pixels in height and width. showed an improvement over the other algorithms, especially for smaller sample sizes. Thus, even this trivial modification can improve performance by several percentage points. Specifically, achieved a better (lower) classification error than all other algorithms besides CNNs for all sample sizes on this real data problem.

5.2 Feature Importance

To evaluate the capability of to identify importance features in manifold-valued data as compared to and . All methods were run on a subset of the MNIST dataset: we only used threes and fives, 100 images from each class.

Figure 4: Averages of images in the two classes and their difference (above). Feature importance from (bottom right) shows less noise than (bottom middle) and is smoother than RF (bottom left).

The feature importance of each pixel is shown in Figure 4. visibly results in a smoother pixel importance, a result most likely from the continuity of neighboring pixels in selected projections. Although SPORF demonstrated empirical improvement of over on the MNIST data, its projection distribution yields scattered importance of unimportant background pixels as compared to . Since projections in have no continuity constraint, those that select high importance pixels will also select pixels of low importance by chance. This may be a nonissue asymptotically, but is a relevant problem in low sample size settings. , however, shows little or no importance of these background pixels by virtue of the modified projection distribution.

6 Discussion

The success of sparse oblique projections in decision forests has opened up many possible ways to improve axis-aligned decision forests (including random forests and gradient boosting trees) by way of specialized projection distributions. Traditional decision forests have already been applied to structured data, using predefined features to classify images or pixels. Decision forest algorithms like that implemented in the Microsoft Kinect showed great success but ignore pixel continuity and specialize for a specific data modality, namely images [kinect_rf].

We expanded upon sparse oblique projections and introduced a structural projection distribution that uses prior knowledge of the topology of a feature space. The open source implementation of ²²2https://neurodata.io/sporf/ has allowed for a relatively easy implementation of , creating a flexible classification method for a variety of data modalities. We showed in various simulated settings that appropriate domain knowledge can improve the projection distribution to yield impressive results that challenge the strength of deep learning techniques on manifold-valued data. On the MNIST data set showed modest improvements over the other algorithms besides CNNs and smoother importance plots than the other decision forest algorithms. This is in spite of the the data set’s low resolution images which are harder for the modified projection distributions to take advantage of.

Research into other, task-specific convolution kernels may lead to improved results in real-world computer vision tasks. Such structured projection distributions, while incorporated into here, may also be incorporated into other state of the art algorithms such as XgBoost

[XGboost].

References

7 Appendix

1:(1)

D_{n}

: training data (2)

d

: dimensionality of the projected space, (3)

f_{A}

: distribution of the atoms, (4)

Θ

: set of split eligibility criteria

2:A decision tree

T

3:function

T =

growtree(

X, y, f_{A}, Θ

)

c = 1

▹

c

is the current node index

M = 1

▹

M

is the number of nodes currently existing

S^{(c)} =

bootstrap(

{1, . . ., n}

)

▹

S^{(c)}

is the indices of the observations at node

c

7: while

c < M + 1

▹

visit each of the existing nodes

(X^{'}, y^{'}) = (x_{i}, y_{i})_{i \in S^{(c)}}

▹

data at the current node

k = 1, \dots, K

n_{k}^{(c)} = \sum_{i \in S^{(c)}} I [y_{i} = k]

▹

class counts (for classification)

9: if

Θ

satisfied then

▹

do we split this node?

10:

A = [a_{1} \dots a_{d}] \sim f_{A}

▹

sample random

p \times d

matrix of atoms

11:

˜ X = A^{T} X^{'} = ({˜ x}_{i})_{i \in S^{(c)}}

▹

random projection into new feature space

12:

(j^{*}, t^{*}) =

findbestsplit(

˜ X, y^{'}

)

▹

Algorithm 2

13:

S^{(M + 1)} = {i : a_{j^{*}} \cdot {˜ x}_{i} \leq t^{*} \forall i \in S^{(c)}}

▹

assign to left child node

14:

S^{(M + 2)} = {i : a_{j^{*}} \cdot {˜ x}_{i} > t^{*} \forall i \in S^{(c)}}

▹

assign to right child node

15:

a^{* (c)} = a_{j^{*}}

▹

store best projection for current node

16:

τ^{* (c)} = t^{*}

▹

store best split threshold for current node

17:

κ^{(c)} = {M + 1, M + 2}

▹

node indices of children of current node

18:

M = M + 2

▹

update the number of nodes that exist

19: else

20:

(a^{* (c)}, τ^{* (c)}, κ^{* (c)}) =

NULL

21: end if

22:

c = c + 1

▹

move to next node

23: end while

24: return

(S^{(1)}, {a^{* (c)}, τ^{* (c)}, κ^{(c)}, {n_{k}^{(c)}}_{k \in Y}}_{c = 1}^{m - 1})

25:end function

Algorithm 1 Learning a Manifold Forest decision tree.

1:(1)

(X,y)∈\Realp×n×Yn

, where

Y = {1, \dots, K}

2:(1) dimension

j^{*}

, (2) split value

τ^{*}

3:function

(j^{*}, τ^{*}) =

findbestsplit(

X, y

)

4: for

j = 1, \dots, p

5: Let

x^{(j)} = (x_{1}^{(j)}, \dots, x_{n}^{(j)})

be the

j t h

row of

X

{m_{i}^{j}}_{i \in [n]} =

sort(

x^{(j)}

)

▹

m_{i}^{j}

is the index of the

i^{t h}

smallest value in

x^{(j)}

t = 0

▹

initialize split to the left of all observations

n^{'} = 0

▹

number of observations left of the current split

n^{''} = n

▹

number of observations right of the current split

10: if (task is classification) then

11: for

k = 1, \dots, K

12:

n_{k} = \sum_{i = 1}^{n} I [y_{i} = k]

▹

total number of observations in class

k

13:

n_{k}^{'} = 0

▹

number of observations in class

k

left of the current split

14:

n_{k}^{''} = n_{k}

▹

number of observations in class

k

right of the current split

15: end for

16: end if

17: for

t = 1, \dots, n - 1

▹

assess split location, moving right one at a time

18:

({(n_{k}^{'}, n_{k}^{''})}, n^{'}, n^{''}, y_{m_{t}^{j}}) =

increment(

{(n_{k}^{'}, n_{k}^{''})}, n^{'}, n^{''}, y_{m_{t}^{j}}

)

19:

Q^{(j, t)} =

score(

{(n_{k}^{'}, n_{k}^{''})}, n^{'}, n^{''}

)

▹

measure of split quality

20: end for

21: end for

22:

i = 0, 1

c_{i} = m_{t^{*} + i}^{j^{*}}

23:

τ^{*} = \frac{1}{2} (x_{c_{0}}^{(j^{*})} + x_{c_{1}}^{(j^{*})})

▹

compute the actual split location from the index

j^{*}

24: return

(j^{*}, τ^{*})

25:end function

Algorithm 2 Finding the best node split. This function is called by growtree (Alg 1) at every split node. For each of the

p

dimensions in

X∈\Realp×n

, a binary split is assessed at each location between adjacent observations. The dimension

j^{*}

and split value

τ^{*}

j^{*}

that best split the data are selected. The notion of “best” means maximizing some choice in scoring function. In classification, the scoring function is typically the reduction in Gini impurity or entropy. The increment function called within this function updates the counts in the left and right partitions as the split is incrementally moved to the right.

Notes

¹ Department of Biomedical Engineering, Johns Hopkins University
¹ Department of Biomedical Engineering, Johns Hopkins University
¹ Department of Biomedical Engineering, Johns Hopkins University
¹ Department of Biomedical Engineering, Johns Hopkins University
¹ Department of Biomedical Engineering, Johns Hopkins University
² Center for Imaging Science, Johns Hopkins University
³ Kavli Neuroscience Discovery Institute, Johns Hopkins University

Manifold Forests: Closing the Gap on Neural Networks

Authors

Towards Robust Classification with Deep Generative Forests

Random Hinge Forest for Differentiable Learning

MVC-Net: A Convolutional Neural Network Architecture for Manifold-Valued Images With Applications

One Class Splitting Criteria for Random Forests

Robust Similarity and Distance Learning via Decision Forests

That looks interesting! Personalizing Communication and Segmentation with Random Forest Node Embeddings

Machine Learning Calabi-Yau Four-folds

1 Introduction

2 Background and Related Work

2.1 Classification

2.2 Random Forests

2.3 Sparse Projection Oblique Randomer Forests

3 Methods

3.1 Random Projection Forests on Manifolds

3.2 Feature Importance

4 Simulation Results

4.1 Simulation Settings

4.2 Classification Accuracy

4.3 Run Time

5 Real Data Results

5.1 Classification Accuracy

5.2 Feature Importance

6 Discussion

References

7 Appendix

Notes

Manifold Forests: Closing the Gap on Neural Networks

Authors

Towards Robust Classification with Deep Generative Forests

Random Hinge Forest for Differentiable Learning

MVC-Net: A Convolutional Neural Network Architecture for Manifold-Valued Images With Applications

One Class Splitting Criteria for Random Forests

Robust Similarity and Distance Learning via Decision Forests

That looks interesting! Personalizing Communication and Segmentation with Random Forest Node Embeddings

Machine Learning Calabi-Yau Four-folds

This week in AI

1 Introduction

2 Background and Related Work

2.1 Classification

2.2 Random Forests

2.3 Sparse Projection Oblique Randomer Forests

3 Methods

3.1 Random Projection Forests on Manifolds

3.2 Feature Importance

4 Simulation Results

4.1 Simulation Settings

4.2 Classification Accuracy

4.3 Run Time

5 Real Data Results

5.1 Classification Accuracy

5.2 Feature Importance

6 Discussion

References

7 Appendix

Notes