Robust Model Compression Using Deep Hypotheses

03/13/2021
by   Omri Armstrong, et al.
0

Machine Learning models should ideally be compact and robust. Compactness provides efficiency and comprehensibility whereas robustness provides resilience. Both topics have been studied in recent years but in isolation. Here we present a robust model compression scheme which is independent of model types: it can compress ensembles, neural networks and other types of models into diverse types of small models. The main building block is the notion of depth derived from robust statistics. Originally, depth was introduced as a measure of the centrality of a point in a sample such that the median is the deepest point. This concept was extended to classification functions which makes it possible to define the depth of a hypothesis and the median hypothesis. Algorithms have been suggested to approximate the median but they have been limited to binary classification. In this study, we present a new algorithm, the Multiclass Empirical Median Optimization (MEMO) algorithm that finds a deep hypothesis in multi-class tasks, and prove its correctness. This leads to our Compact Robust Estimated Median Belief Optimization (CREMBO) algorithm for robust model compression. We demonstrate the success of this algorithm empirically by compressing neural networks and random forests into small decision trees, which are interpretable models, and show that they are more accurate and robust than other comparable methods. In addition, our empirical study shows that our method outperforms Knowledge Distillation on DNN to DNN compression.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/19/2017

A remark on "Robust machine learning by median-of-means"

We explore the recent results announced in "Robust machine learning by m...
research
09/23/2022

Another look at halfspace depth: Flag halfspaces with applications

The halfspace depth is a well studied tool of nonparametric statistics i...
research
08/18/2022

Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification

Compared to traditional learning from scratch, knowledge distillation so...
research
12/03/2019

Training Robust Tree Ensembles for Security

Tree ensemble models including random forests and gradient boosted decis...
research
12/28/2018

Improving the Interpretability of Deep Neural Networks with Knowledge Distillation

Deep Neural Networks have achieved huge success at a wide spectrum of ap...
research
04/10/2019

Network depth: identifying median and contours in complex networks

Centrality descriptors are widely used to rank nodes according to specif...
research
02/10/2019

Adversarially Trained Model Compression: When Robustness Meets Efficiency

The robustness of deep models to adversarial attacks has gained signific...

Please sign up or login with your details

Forgot password? Click here to reset