A better measure of relative prediction accuracy for model selection and model estimation

by   Chris Tofallis, et al.

Surveys show that the mean absolute percentage error (MAPE) is the most widely used measure of forecast accuracy in businesses and organizations. It is however, biased: When used to select among competing prediction methods it systematically selects those whose predictions are too low. This is not widely discussed and so is not generally known among practitioners. We explain why this happens. We investigate an alternative relative accuracy measure which avoids this bias: the log of the accuracy ratio: log (prediction / actual). Relative accuracy is particularly relevant if the scatter in the data grows as the value of the variable grows (heteroscedasticity). We demonstrate using simulations that for heteroscedastic data (modelled by a multiplicative error factor) the proposed metric is far superior to MAPE for model selection. Another use for accuracy measures is in fitting parameters to prediction models. Minimum MAPE models do not predict a simple statistic and so theoretical analysis is limited. We prove that when the proposed metric is used instead, the resulting least squares regression model predicts the geometric mean. This important property allows its theoretical properties to be understood.



page 1

page 2

page 3

page 4


Regression model selection via log-likelihood ratio and constrained minimum criterion

Although the log-likelihood is widely used in model selection, the log-l...

Assessing the Significance of Model Selection in Ecology

Model Selection is a key part of many ecological studies, with Akaike's ...

Mean Absolute Percentage Error for regression models

We study in this paper the consequences of using the Mean Absolute Perce...

Estimating model evidence using ensemble-based data assimilation with localization - The model selection problem

In recent years, there has been a growing interest in applying data assi...

An Exponential Factorization Machine with Percentage Error Minimization to Retail Sales Forecasting

This paper proposes a new approach to sales forecasting for new products...

Evaluating prediction systems in software project estimation

Context: Software engineering has a problem in that when we empirically ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.