Statistical Estimation of Malware Detection Metrics in the Absence of Ground Truth

09/24/2018
by   Pang Du, et al.
0

The accurate measurement of security metrics is a critical research problem because an improper or inaccurate measurement process can ruin the usefulness of the metrics, no matter how well they are defined. This is a highly challenging problem particularly when the ground truth is unknown or noisy. In contrast to the well perceived importance of defining security metrics, the measurement of security metrics has been little understood in the literature. In this paper, we measure five malware detection metrics in the absence of ground truth, which is a realistic setting that imposes many technical challenges. The ultimate goal is to develop principled, automated methods for measuring these metrics at the maximum accuracy possible. The problem naturally calls for investigations into statistical estimators by casting the measurement problem as a statistical estimation problem. We propose statistical estimators for these five malware detection metrics. By investigating the statistical properties of these estimators, we are able to characterize when the estimators are accurate, and what adjustments can be made to improve them under what circumstances. We use synthetic data with known ground truth to validate these statistical estimators. Then, we employ these estimators to measure five metrics with respect to a large dataset collected from VirusTotal. We believe our study touches upon a vital problem that has not been paid due attention and will inspire many future investigations.

READ FULL TEXT
research
11/30/2020

RealCause: Realistic Causal Inference Benchmarking

There are many different causal effect estimators in causal inference. H...
research
09/23/2021

A Framework for Cluster and Classifier Evaluation in the Absence of Reference Labels

In some problem spaces, the high cost of obtaining ground truth labels n...
research
09/03/2021

Relating the Partial Dependence Plot and Permutation Feature Importance to the Data Generating Process

Scientists and practitioners increasingly rely on machine learning to mo...
research
05/26/2021

The statistical advantage of automatic NLG metrics at the system level

Estimating the expected output quality of generation systems is central ...
research
03/05/2021

Measuring Model Biases in the Absence of Ground Truth

Recent advances in computer vision have led to the development of image ...
research
02/09/2021

Benchmarks, Algorithms, and Metrics for Hierarchical Disentanglement

In representation learning, there has been recent interest in developing...
research
11/28/2019

Towards Reliable Evaluation of Road Network Reconstructions

Existing performance measures rank delineation algorithms inconsistently...

Please sign up or login with your details

Forgot password? Click here to reset