Robustness Verification of Tree-based Models

by   Hongge Chen, et al.

We study the robustness verification problem for tree-based models, including decision trees, random forests (RFs) and gradient boosted decision trees (GBDTs). Formal robustness verification of decision tree ensembles involves finding the exact minimal adversarial perturbation or a guaranteed lower bound of it. Existing approaches find the minimal adversarial perturbation by a mixed integer linear programming (MILP) problem, which takes exponential time so is impractical for large ensembles. Although this verification problem is NP-complete in general, we give a more precise complexity characterization. We show that there is a simple linear time algorithm for verifying a single tree, and for tree ensembles, the verification problem can be cast as a max-clique problem on a multi-partite graph with bounded boxicity. For low dimensional problems when boxicity can be viewed as constant, this reformulation leads to a polynomial time algorithm. For general problems, by exploiting the boxicity of the graph, we develop an efficient multi-level verification algorithm that can give tight lower bounds on the robustness of decision tree ensembles, while allowing iterative improvement and any-time termination. OnRF/GBDT models trained on 10 datasets, our algorithm is hundreds of times faster than the previous approach that requires solving MILPs, and is able to give tight robustness verification bounds on large GBDTs with hundreds of deep trees.


page 1

page 2

page 3

page 4


An Efficient Adversarial Attack for Tree Ensembles

We study the problem of efficient adversarial attacks on tree based ense...

On ℓ_p-norm Robustness of Ensemble Stumps and Trees

Recent papers have demonstrated that ensemble stumps and trees could be ...

Versatile Verification of Tree Ensembles

Machine learned models often must abide by certain requirements (e.g., f...

Robust Decision Trees Against Adversarial Examples

Although adversarial examples and model robustness have been extensively...

Training Robust Tree Ensembles for Security

Tree ensemble models including random forests and gradient boosted decis...

Linear TreeShap

Decision trees are well-known due to their ease of interpretability. To ...

Formal Verification of Input-Output Mappings of Tree Ensembles

Recent advances in machine learning and artificial intelligence are now ...

1 Introduction

Recent studies have demonstrated that neural network models are vulnerable to adversarial perturbations—a small and imperceptible-to-human input perturbation can easily change the predicted label 

[31, 15, 6, 13]

. This has created serious security threats to many real applications so it becomes important to formally verify the robustness of machine learning models. Usually, the robustness verification problem can be cast as finding the minimal adversarial perturbation to an input example that can change the predicted class label. A series of robustness verification algorithms have been developed for neural network models 

[19, 32, 36, 35, 34, 38, 14, 29]

, where efficient algorithms are mostly based on relaxation or approximation of nonlinear activation functions of neural networks.

We study the robustness verification problem of tree-based models, including a single decision tree and tree ensembles such as random forests (RFs) and gradient boosted decision trees (GBDTs). These models have been widely used in practice and recent studies have demonstrated that both RFs and GBDTs are vulnerable to adversarial perturbations [18, 12, 8]. It is thus important to develop a formal robustness verification algorithm for tree-based models. Robustness verification requires computing the minimal adversarial perturbation. [18] showed that computing minimal adversarial perturbation for tree ensemble is NP-complete in general, and they proposed a Mixed-Integer Linear Programming (MILP) approach to compute the minimal adversarial perturbation. Although exact verification is NP-hard, in order to have an efficient verification algorithm for real applications we seek to answer the following questions:

  • [noitemsep,topsep=0pt,parsep=0pt,partopsep=0pt]

  • Can we have efficient polynomial time algorithms for exact verification under some special circumstances?

  • For general tree ensemble models with a large number of trees, can we efficiently compute a meaningful lower bounds on robustness while scaling to large tree ensembles?

In this paper, we answer the above-mentioned questions in the affirmative by formulating the verification problem of tree ensemble as a graph problem. First, we show that for a single decision tree, robustness verification can be done exactly in linear time. Then we show that for an ensemble of trees, the verification problem is equivalent to finding the maximum cliques in a -partite graph, and the graph is in a special form with boxicity equal to the input feature dimension. Therefore, for low-dimensional problems, verification can also be done in polynomial time with maximum clique searching algorithms. Finally, for large-scale tree ensembles, we propose a multiscale verification algorithm by exploiting the boxicity of the graph, which can give tight lower bounds on robustness. Furthermore, it supports any-time termination: we can stop the algorithm at any time to obtain a reasonable lower bound given a computation time constraint. Our proposed algorithm is efficient and is scalable to large tree ensemble models. For instance, on a large multi-class GBDT with 200 trees robustly trained on the MNIST dataset (using [8]), we obtained 78% verified robustness accuracy on test set with maximum perturbation and the time used for verifying each test example is 12.6 seconds, whereas the MILP method uses around 10 min for each test example.

2 Background and Related Work

Adversarial Robustness

For simplicity, we consider a multi-class classification model where is the input dimension and is number of classes. For an input example , assuming that is the correct label, the minimal adversarial perturbation is defined by


Note that we focus on the norm measurement in this paper which is widely used in recent studies [22, 36, 5]. Exactly solving (1) is usually intractable. For example, if is a neural network, (1) is non-convex and [19] showed that solving (1

) is NP-complete for ReLU networks.

Adversarial attacks are algorithms developed for finding a feasible solution of (1), where is an upper bound of . Many algorithms have been proposed for attacking machine learning models  [15, 20, 6, 22, 9, 10, 16, 3, 12, 24, 21]. Most practical attacks cannot reach the minimal adversarial perturbation due to the non-convexity of (1). Therefore, attacking algorithms cannot provide any formal guarantee on model robustness [1, 33].

On the other hand, robustness verification algorithms are designed to find the exact value or a lower bound of . An exact verifier needs to solve (1) to the global optimal, so typically we resort to relaxed verifiers that give lower bounds. When a verification algorithm finds a lower bound , it guarantees that no adversarial example exists within a radius ball around . This is important for deploying machine learning algorithms to safety-critical applications such as autonomous vehicles or aircraft control systems [19, 17].

For verification, instead of solving (1) we can also solve the following decision problem of robustness verification


Note that in our setting

. If we can answer this decision problem, a binary search can give us the value of , so the complexity of (2) is in the same order of (1). Furthermore, a safe answer to (2) (always say yes when unsure) will lead to a lower bound of , which is what we want to do in verification. The decision version is also widely used in the verification community since people care about “robustness error at perturbation” which is defined to be the ratio of number of test samples that satisfy (2). Verification methods for neural networks have been studied extensively in the past few years [36, 37, 35, 38, 29, 14, 30, 2].

Adversarial Robustness of Tree-based Models

Unlike neural networks, decision-tree based models are non-continuous step functions, and thus existing neural network verification algorithms cannot be directly applied. To evaluate the robustness of tree-based models, [18] showed that solving (1) for general tree ensemble models is NP-complete, so no polynomial time algorithm can compute unless P=NP. A Mixed Integer Linear Programming (MILP) algorithm was thus proposed in [18] to compute (1) in exponential time. Some hard-label attacking algorithms for neural networks, including the boundary attack [3] and OPT-attack [12], can also be applied since they only require function evaluation of the non-smooth (hard-label) decision function , and can be viewed as faster ways to compute an upper bound of . To the best of our knowledge, there is no prior existing algorithm for efficient verification, or equivalently, efficiently computing a lower bound of for ensemble trees.

3 Proposed Algorithm

In this section, we propose the first-ever tree ensemble verification algorithms. The tree ensemble exact verification problem is NP-complete by its nature, and here we propose a series of efficient verification algorithms for real applications. First, we will introduce a linear time algorithm for exactly computing the minimal adversarial distortion for verifying a single decision tree. For an ensemble of trees, we cast the verification problem into a max-clique searching problem in K-partite graphs. For large-scale tree ensembles, we then propose an efficient multi-level algorithm for verifying an ensemble of decision trees.

3.1 Exactly Verifying a Single Tree in Linear Time

Although computing for a tree ensemble is NP-complete [18], we show that a linear time algorithm exists for finding the minimum adversarial perturbation and computing for a single decision tree. We assume the decision tree has nodes and the root node is indexed as . For a given example with features, starting from the root, traverses the decision tree model until reaching a leaf node. Each internal node, say node , has two children and a feature-threshold pair to determine the traversal direction— will be passed to the left child if and to the right child otherwise. Each leaf node has a value corresponding to the predicted class label for a classification tree, or a real value for a regression tree.

Conceptually, the main idea of our single tree verification algorithm is to compute a -dimensional box for each leaf node such that any example in this box will fall into this leaf. Mathematically, the node ’s box is defined as the Cartesian product of closed intervals on the real line. By definition, the root node has box and given the box of an internal node , its children’s boxes can be obtained by changing only one interval of the box based on the split condition . More specifically, if are node ’s left and right child node respectively, then we set their boxes and by setting


After computing the boxes for internal nodes, we can also obtain the boxes for leaf nodes using (3). Therefore computing the boxes for all the leaf nodes of a decision tree can be done by a depth-first search traversal of the tree with time complexity .

With the boxes computed for each leaf node, the minimum perturbation required to change to go to a leaf node

can be written as a vector

defined as


Then the minimal distortion can be computed as , where is the original label of , and is the label for leaf node . To find , we check for all leaves and choose the smallest perturbation. This is a linear-time algorithm for exactly verifying the robustness of a single decision tree.

In fact, this time algorithm is used to illustrate the concept of “boxes” that will be used later on for the tree ensemble case. If our final goal is to verify a single tree, we can have a more efficient algorithm by combining the distance computation (4) in the tree traversal procedure, and the resulting algorithm will take only time. This algorithm is presented as Algorithm 1 in the appendix.

3.2 Verifying Tree Ensembles by Max-clique Enumeration

Now we discuss the robustness verification for tree ensembles. Assuming the tree ensemble has decision trees, we use to denote the set of leaf nodes of tree and to denote the function that maps the input example to the leaf node of tree according to its traversal rule. Given an input example , the tree ensemble will pass to each of these trees independently and reaches leaf nodes for all . Each leaf node will assign a prediction value . For simplicity we start with the binary classification case, with ’s original label being and we want to turn it into . For binary classification the prediction of the tree ensemble is computed by , which covers both GBDTs and random forests, two widely used tree ensemble models. Assume has the label , that means for , and our task is to verify if the sign of the summation can be flipped within .

We consider the decision problem of robustness verification (2). A naive analysis will need to check all the points in which is uncountably infinite. To reduce the search space to finite, we start by defining some notation: let to be all the possible tuples of leaf nodes and let be the function that maps to the corresponding leaf nodes. Therefore, a tuple directly determines the model prediction . Now we define a valid tuple for robustness verification:

Definition 1.

A tuple is valid if and only if there exists an such that .

The decision problem of robustness verification (2) can then be written as:

Does there exist a valid tuple such that ?

Next, we show how to model the set of valid tuples. We have two observations. First, if a tuple contains any node with , then it will be invalid. Second, there exists an such that if and only if , or equivalently:

We show that the set of valid tuples can be represented as cliques in a graph , where and . In this graph, nodes are the leaves of all trees and we remove every leaf that has empty intersection with . There is an edge between node and if and only if their boxes intersect. The graph will then be a -partite graph since there cannot be any edge between nodes from the same tree, and thus maximum cliques in this graph will have nodes. We define each part of the -partite graph as . Here a “part” means a disjoint and independent set in the -partite graph. The following lemma shows that intersections of boxes have very nice properties:

Lemma 1.

For boxes , if for all then and their intersection will also be an nonempty box: .

The proof can be found in the appendix. Based on the above lemma, each -clique (fully connected subgraph with nodes) in can be viewed as a set of leaf nodes that has nonempty intersection with each other and also have nonempty intersection with , so the intersection of those boxes and will be a nonempty box, which implies each -clique corresponds to a valid tuple of leaf nodes:

Lemma 2.

A tuple is valid if and only if nodes form a -clique (maximum clique) in graph constructed above.

Therefore the robustness verification problem can be formulated as

Is there a maximum clique in such that ? (5)

This reformulation indicates that the tree ensemble verification problem can be solved by an efficient maximum clique enumeration algorithm. Some standard maximum clique searching algorithms can be applied here to perform verification:

  • [noitemsep,topsep=0pt,parsep=0pt,partopsep=0pt,leftmargin=*]

  • Finding -cliques in -partite graphs: Any algorithm for finding all the maximum cliques in can be used. The classic B-K backtracking algorithm [4] takes time to find all the maximum cliques where is the number of nodes in . Furthermore, since our graph is a -partite graph, we can apply some specialized algorithms designed for finding all the -cliques in -partite graphs [23, 25, 28].

  • Polynomial time algorithms exist for low-dimensional problems: Another important property for graph is that each node in is a -dimensional box and each edge indicates intersection of two boxes. This implies our graph is with “boxicity ” (see [7] for detail).  [7] proved that the number of maximum cliques will only be and it is able to find the maximum weight clique in time. Therefore, for problems with very small , the time complexity for verification is actually polynomial.

Therefore we can exactly solve the tree ensemble verification problem using algorithms for maximum cliques searching in -partite graph, and its time complexity is found to be as follows:

Theorem 1.

Exactly verifying the robustness of a -tree ensemble with at most leaves per tree and dimensional features takes time.

This is a direct consequence of the fact that the number of -cliques in a -partite graph with vertices per part is bounded by , and number of maximum cliques in a graph with a total of vertices with boxicity is . For a general graph, since and can be in and  [27], it can still be exponential. But the theorem gives a more precise characterization for the complexity of the verification problem for tree ensembles.

Based on the nice properties of maximum cliques searching problem, we propose a simple and elegant algorithm that enumerates all -cliques on a -partite graph with a known boxicity in Algorithm 2 in the appendix, and we can use this algorithm for tree ensemble verification when the number of trees or the dimension of features is small.

3.3 An Efficient Multi-level Algorithm for Verifying the Robustness of a Tree Ensemble

Figure 1: The proposed multi-level verification algorithm. Lines between leaf node i on tree and leaf node on indicate that their feature boxes intersect (i.e., there exists an input such that tree 1 predicts and tree 2 predicts ).

Practical tree ensembles usually have tens or hundreds of trees and with large dimensions, so Algorithm 2 will take exponential time and will be too slow. We thus develop an efficient multi-level algorithm for computing verification bounds by further exploiting the boxicity of the graph.

Figure 1 illustrates the graph and how our multilevel algorithm runs. There are four trees and each tree has four leaf nodes. A node is colored if it has nonempty intersection with ; uncolored nodes are discarded. To answer question (5), we need to compute the maximum among all -cliques, denoted by . As mentioned before, for robustness verification we only need to compute an upper bound of in order to get a lower bound of minimal adversarial robustness. In the following, we will first discuss algorithms for computing an upper bound at the top level, and then show how our multi-scale algorithm iterative refines this bound until reaching the exact solution .

Bounds for a single level.

To compute an upper bound of , a naive approach is to assume that the graph is fully connected between independent sets (fully connected -partite graph) and in this case the maximum sum of node values will be the sum of the maximum value of each independent set:


One can easily show this is an upper bound of since any original -clique will still be considered when we add more edges to the graph.

Another slightly better approach is to exploit the edge information but only between tree and . If we search over all the length- paths from the first to the last group and define the value of a path to be , then the maximum valued path will be a upper bound of . This can be computed in linear time using dynamic programming. We scan nodes from tree to tree , and for each node we store a value which is the maximum value of paths from tree to this node. At tree and node , the value can be computed by


Taking the max value in the last tree will give us a tighter upper bound of .

Merging independent sets

Now we try to refine our bound. Our approach is to partition the graph into groups, each with independent sets. Within each group, we find all the -cliques and use a new “pseudo node” to represent each -clique. -cliques in a -partite graph can be enumerated efficiently if we choose to be a relative small number (e.g., or in the experiments).

Now we exploit the boxicity property of our graph to form a graph among these cliques (illustrated as the second level nodes in Figure 1). By Lemma 1, we know that the intersection of boxes will still be a box, so each -clique is still a box and can be represented as a pseudo node in the level-2 graph. Also because each pseudo node is still a box, we can easily form edges between pseudo nodes to indicate the nonempty overlapping between them and this will be a -partite boxicity graph since no edge can be formed for the cliques within the same group. Thus we get the level-2 graph. With the level-2 graph, we can again run the single level algorithm to compute a upper bound on to get a lower bound of in (1), but different from the level-1 graph, now we already considered all the within-group edges so the value we get will be less or equal to the level-1 bound, which means tighter.

The overall multi-level framework

We can run the algorithm level by level until merging all the groups into one, and in the final level the pseudo nodes will correspond to the -cliques in the original graph, and the maximum value will be exactly . Therefore, our algorithm can be viewed as an anytime algorithm that refines the upper bound level-by-level until reaching the maximum value. Although getting to the final level still requires exponential time, in practice we can stop at any level (denoted as ) and get a reasonable bound. In experiments, we will show that by merging few trees we already get a bound very close to the final solution. Algorithm 3 in the appendix gives the complete procedure.

Handling multi-class tree ensembles

For a multiclass classification problem, say -class classification problem, groups of tree ensembles and each with trees are built for the classification task; for the -th tree in group , prediction outcome is denoted as where is the function that maps the input example to a leaf node of tree in group . The final prediction is given by . Given an input example with ground-truth class and an attack target class , we extract trees for class and class , and flip the sign of all prediction values for trees in group , such that initially

for a correctly classified example. Then, we are back to the binary case with

trees, and can still apply our multi-level framework to obtain a lower bound of for this target attack pair . Robustness of an untargeted attack can be evaluated by taking .

3.4 Verification Problems Beyond Ordinary Robustness

The above discussions focus on the decision problem of robustness verification (2). In fact, our approach works for a more general verification problem for any -dimensional box :

Is there any such that ? (8)

In typical robustness verification settings, is defined to be but in fact we can allow any boxes in our algorithm. For a general , Lemma 1 still holds so all of our algorithms and analysis can go through. The only thing we need to change is to compute the intersection between and each box of leaf node at the first level in Figure 1 to eliminate nodes that have empty intersection with . So robustness verification is just a special case where we remove all the nodes with empty intersection with . For example, if we want to identify a set of unimportant variables, where any change in cannot alter the prediction for a given sample , then we can choose as if and otherwise. Similarly, we can also compute a set of anchor features (similar to [26]) such that once a set of features are fixed, any perturbation outside the set cannot change the prediction.

4 Experiments

We evaluate our proposed method for ensemble tree robustness verification on two tasks: binary and multiclass classification on 10 public datasets including both small and large scale datasets111

Our code (XGBoost compatible) is available at The statistics of the data sets are shown in the appendix. As we defined in Section 2, is the radius of minimum adversarial perturbation that reflects true model robustness, but is hard to obtain; our method finds that is a lower bound of , which guarantees that no adversarial example exists within radius . A high quality lower bound should be close to . We include the following algorithms in our comparisons:

  • [noitemsep,topsep=0pt,parsep=0pt,partopsep=0pt,leftmargin=*]

  • Cheng’s attack [12] provides results on adversarial attacks on these models, which gives an upper bound of the model robustness . We denote it as and .

  • MILP: an MILP based method [18] gives the exact . MILP is proposed for adversarial attacks on tree ensembles and thus always holds for all lower bounds. MILP needs to solve a Mixed Integer Linear Program to find and runs in exponential time, and is slow when the number of trees or dimension of the features increases.

  • LP relaxed MILP: a Linear Programming (LP) relaxed MILP formulation by directly changing all binary variables to continuous ones. Since the constraints are removed, solving the minimization of MILP gives a lower bound of robustness,

    , serving as a baseline method.

  • Our proposed multi-level verification framework in Section 3.3 (with pseudo code as Algorithm 3 in the appendix). We are targeting to compute robustness interval for tree ensemble verification.

In Tables 1 and 2 we show empirical comparisons on these 10 datasets. We consider robustness, and normalize our datasets to such that perturbations on different datasets are comparable. We include both standard (naturally trained) GBDT models (Table 1) and robust GBDT models (Table  2) in [8]. The robust GBDTs try to optimize the model performance under the worst-case perturbation of input features, which leads to a max-min saddle point problem when finding the split at each node. All GBDTs are implemented under XGBoost framework [11]. The number of trees in GBDT and parameters used in training GBDT for different datasets are shown in Table 3 in the appendix. Because we solve the decision problem of robustness verification, we use 10 times binary search to find the largest in all experiments, and the reported time is the total time including all binary search trials. We present the average of or over 500 examples. The MILP based method from [18] is an accurate but very slow method; the results marked with an asterisk (“*”) in the table have very long running time and thus we only evaluate 50 examples instead of 500.



Cheng’s attack [12]   MILP  [18]   LP relaxation   Ours   Ours vs. MILP
avg. avg. time   avg. avg. time   avg. avg. time   avg. avg. time   speedup


.221 2.18s   .210 .012s   .064 .009s   2 1 .208 .001s   .99 12X
covtype   .058 4.76s   s   s   2 3 .022 3.39s   .79 105X
cod-rna   .054 2.13s   .035 .485s   .017 .222s   2 3 .033 .059s   .94 8.2X
diabetes   .064 1.70s   .049 .061s   .015 .026s   3 2 .042 .018s   .86 3.4X
Fashion-MNIST   .048 12.2s   s   s   2 1 .012 11.8s   .86 97X
HIGGS   .015 3.80s   min   min   4 1 .0022 1.29s   .79 3163X
ijcnn1   .047 2.72s   .030 4.64s   .008 2.67s   2 2 .026 .101s   .87 4.6X
MNIST   .070 11.1s   s   s   2 2 .011 5.14s   1.00 71X

.027 5.83s   .00076 47.2s   .0002 39.7s   2 1 .0005 .404s   .66 117X
MNIST 2 vs. 6   .152 12.0s   .057 23.0s   .016 11.6s   4 1 .046 .585s   .81 39X


Table 1: Average distortion over 500 examples and average verification time per example for three verification methods. Here we evaluate the bounds for standard (natural) GBDT models. Results marked with an asterisk (“*”) are the averages of 50 examples due to long running time. is the number of independent sets and is the number of levels in searching cliques used in our algorithm. A ratio close to 1 indicates better lower bound quality.



Cheng’s attack [12]   MILP  [18]   LP relaxation   Ours   Ours vs. MILP
avg. avg. time   avg. avg. time   avg. avg. time   avg. avg. time   speedup


.404 1.96s   .400 .009s   .078 .008s   2 1 .399 .001s   1.00 9X
covtype   .079 .481s   s   s   2 3 .032 4.84s   .70 63X
cod-rna   .062 2.02s   .055 .607s   .017 .410s   2 3 .052 .104s   .95 5.8X
diabetes   .137 1.52s   .112 .034s   .035 .013s   3 2 .109 .006s   .97 5.7X
Fashion-MNIST   .153 13.9s   min   min   2 1 .071 18.0s   .78 137X
HIGGS   .023 3.58s   min   min   4 1 .0063 1.41s   .75 2511X
ijcnn1   .054 2.63s   .036 2.52s   .009 1.26s   2 2 .032 0.58s   .89 4.3X
MNIST   .367 1.41s   s   s   2 2 .253 12.6s   .96 49X

.048 4.97s   .015 83.7s   .0024 60.4s   2 1 .011 .345s   .73 243X
MNIST 2 vs. 6   .397 17.2s   .313 91.5s   .039 40.0s   4 1 .308 3.68s   .98 25X


Table 2: Average distortion over 500 examples and average verification time per example for three verification methods and an attack method. Here we evaluate the bounds for robustly trained models introduced in [8]. Results marked with an asterisk (“*”) are the averages of 50 examples due to long running time. is the number of independent sets and is the number of levels in searching cliques used in our algorithm. A ratio close to 1 indicates better lower bound quality.

From Tables 1 and 2 we can see that our method gives a tight lower bound compared to from MILP, while achieving up to X speedup on large models. The running time of the baseline LP relaxation, however, is on the same order of magnitude as the MILP method, but the results are much worse, with . Our proposed method given as Algorithm 3 is a multi-scale any-time approach, that is, it can stop at any scale to get a reasonable robustness bound. Figure 3 shows how the tightness of our robustness verification lower bounds changes with different size of clique per level () and different number of levels (). We test on a 20-tree standard GBDT model on the diabetes dataset. Here we also show the exact bound by the MILP method. Our verification bound converges to the MILP bound as more levels of clique enumerations are used. Also, when we find larger cliques within each level, the bound becomes tighter.

To show the scalability of our method, we vary the number of trees in GBDT and compare per example running time with MILP method on ijcnn1 dataset in Figure 3. We see that our proposed methods spend much less time on each example compared to the MILP method and their running time grows slower than MILP when the number of trees increases.

Figure 2: Robustness bounds obtained with different parameters (, ) on a 20-tree standard GBDT model with the diabetes dataset (left) and a 20-tree robust GBDT model with the ijcnn1 dataset (right). converges to as increases.
Figure 3: Running time of MILP and our method on GBDT model with different number of trees on the ijcnn1 dataset.

In Section 3.4, we showed that the proposed algorithm works for a more general verification problem such as how to identify unimportant features in the data, where any change on those features cannot change the prediction. Here we use MNIST data as an example for this task where we perturb pixels (features) on MNIST test images on both standard and robustly trained multi-class decision trees with depth 8. In Figure 4, yellow pixels cannot change prediction for any perturbation and a darker pixel represents a smaller lower bound of perturbation to change the model output using that pixel. The standard naturally trained model has some very dark pixels compared to the robust model.

Figure 4: Identifying important pixels on 3 MNIST images. Left: digit image; Middle: standard GBDT model; Right: robust GBDT model. Changing one of any yellow pixels cannot change prediction; pixels in darker colors tend to affect model prediction more than pixels in lighter colors.


Appendix A Proof of Lemma 1


If we have one dimensional intervals , we want to prove if every pair of them have nonempty overlap . This can be proved by the following. Without loss of generality we assume . For each , implies . Therefore, will be a nonempty set that is contained in . Therefore and it is another interval.

This can be generalized to -dimensional boxes. Assume we have boxes such that for any and . Then for each dimension we can apply the above proof, which implies that and the intersection will be another box. ∎

Appendix B Data Statistics and Model Parameters in Tables 1 and 2

Table 3 presents data statistics and parameters for the models in Tables 1 and 2 in the main text. The standard test accuracy is the model accuracy on natural, unmodified test sets.


training test # of # of # of robust depth standard test acc.
set size set size features classes trees robust natural robust natural


546 137 10 2 4 0.3 8 6 .978 .964
covtype 400,000 181,000 54 7 80 0.2 8 8 .847 .877
cod-rna 59,535 271,617 8 2 80 0.2 5 4 .880 .965
diabetes 614 154 8 2 20 0.2 5 5 .786 .773
Fashion-MNIST 60,000 10,000 784 10 200 0.1 8 8 .903 .903
HIGGS 10,500,000 500,000 28 2 300 0.05 8 8 .709 .760
ijcnn1 49,990 91,701 22 2 60 0.1 8 8 .959 .980
MNIST 60,000 10,000 784 10 200 0.3 8 8 .980 .980
webspam 300,000 50,000 254 2 100 0.05 8 8 .983 .992
MNIST 2 vs. 6 11,876 1,990 784 2 1000 0.3 6 4 .997 .998


Table 3: The data statistics and parameters for the models presented in Tables 1 and 2.

Appendix C An time algorithm for verifying a decision tree

The robustness of a single tree can be easily verified by the following algorithm, which traverse the whole tree and computes the bounding boxes for each node in a depth-first search fashion.

1 Initial ;
2 ComputeRecursive();
3 Function ComputeRecursive()
4        if  is leaf node then
5               if  then
6                     ;
8        else
               /* Checking conditions for the left child */
9               ;
10               ;
11               if  then
12                      if  then
13                            ComputeRecursive(.left_child, )
14                      else
15                             ComputeRecursive(.left_child, ) ;
18               ;
               /* Checking conditions for the right child */
19               ;
20               ;
21               if  then
22                      if  then
23                            ComputeRecursive(.right_child, )
24                      else
25                             ComputeRecursive(.right_child, ) ;
30 end
Algorithm 1 Linear time untargeted attack for a decision tree.

Appendix D Clique Enumeration Algorithm

The algorithm first looks at any first two parts and of the graph and enumerates all 2-cliques in time. Then, each 2-clique found is converted into a “pseudo node” (this is possible due to Lemma 1), and all 2-cliques form a new part of the graph. Then we replace and with , and continue to enumerate all 2-cliques between and to form . A 2-clique between and represents a 3-clique in , and due to boxicity. Note that enumerating all 3-cliques in a general 3-partite graph takes time; thanks to boxicity, our algorithm takes time which equals to only when and form a complete bipartite graph, which is unlikely in common cases. This process continues recursively until we process all parts and have only left, where each vertex in represents a -clique in the original graph. After obtaining all -cliques, we can verify each to compute verification bound.

1 Initial ;
2 for  do
3        ;
        /* is a set of the nodes in a clique. */
        /* Here is a set of 1-cliques. */
5 end for
7 Function CliqueEnumerate()
8        ;
9        for  do
10               ;
11               for  do
12                      for  do
13                             if  then
                                    /* A clique is found, add it as a pseudo node with the intersection of two boxes */
14                                    ;
17                      end for
19               end for
21        end for
22       return ;
24 end
Algorithm 2 Enumerating all -cliques on a -partite graph with a known boxicity

Appendix E Multi-level Verification Framework

Algorithm 3 presents our multi-level decision tree verification framework describe in Section 3.3.

1 Initial ; maximum number of independent sets in a group: ; maximum number of levels: , ;
2 MultiLevelVerify();
3 Function MultiLevelVerify()
4        for  do
5               ;
               /* is a set of the nodes in a clique */
7        end for
8       for  do
9               for  do
10                      CliqueEnumerate()
11               end for
13        end for
14       for  do
15               for  do
16                      sum of the values corresponding to all leaves in .
17               end for
19        end for
20       Run the dynamic programming method in (7) for .
21 end
Algorithm 3 Multi-level verification framework