DeepAI
Log In Sign Up

Probabilistic performance estimators for computational chemistry methods: Systematic Improvement Probability and Ranking Probability Matrix. I. Theory

03/02/2020
by   Pascal Pernot, et al.
0

The comparison of benchmark error sets is an essential tool for the evaluation of theories in computational chemistry. The standard ranking of methods by their Mean Unsigned Error is unsatisfactory for several reasons linked to the non-normality of the error distributions and the presence of underlying trends. Complementary statistics have recently been proposed to palliate such deficiencies, such as quantiles of the absolute errors distribution or the mean prediction uncertainty. We introduce here a new score, the systematic improvement probability (SIP), based on the direct system-wise comparison of absolute errors. Independently of the chosen scoring rule, the uncertainty of the statistics due to the incompleteness of the benchmark data sets is also generally overlooked. However, this uncertainty is essential to appreciate the robustness of rankings. In the present article, we develop two indicators based on robust statistics to address this problem: P_inv, the inversion probability between two values of a statistic, and P_r, the ranking probability matrix. We demonstrate also the essential contribution of the correlations between error sets in these scores comparisons.

READ FULL TEXT

page 1

page 2

page 3

page 4

11/18/2021

Estimating the concentration parameter of a von Mises distribution: a systematic simulation benchmark

In directional statistics, the von Mises distribution is a key element i...
12/01/2022

Prasatul Matrix: A Direct Comparison Approach for Analyzing Evolutionary Optimization Algorithms

The performance of individual evolutionary optimization algorithms is mo...
10/30/2018

A comparison of university performance scores and ranks by MNCS and FSS

In a previous article of ours, we explained the reasons why the MNCS and...
03/27/2013

A Knowledge Engineer's Comparison of Three Evidence Aggregation Methods

The comparisons of uncertainty calculi from the last two Uncertainty Wor...
09/08/2012

Rank Centrality: Ranking from Pair-wise Comparisons

The question of aggregating pair-wise comparisons to obtain a global ran...
05/20/2014

The ROMES method for statistical modeling of reduced-order-model error

This work presents a technique for statistically modeling errors introdu...

Code Repositories

Reproducible-Research

Codes and data to reproduce the results of research by P. Pernot and collaborators


view repo