A General Model Validation and Testing Tool
share
We construct and propose the "Bayesian Validation Metric" (BVM) as a general model validation and testing tool. We find the BVM to be capable of representing all of the standard validation metrics (square error, reliability, probability of agreement, frequentist, area, statistical hypothesis testing, and Bayesian model testing) as special cases and find that it can be used to improve, generalize, or further quantify their uncertainties. Thus, the BVM allows us to assess the similarities and differences between existing validation metrics in a new light. The BVM may be used to select models according to novel model validation comparison measures. We constructed the BVM ratio for the purpose of quantifying model selection under arbitrary definitions of agreement. This construction generalizes the Bayesian model testing framework. As an example of the versatility and effectiveness of our method, we formulated a quantitative comparison function to represent the visual inspection an engineer might use to validate a model. The BVM ratio leads to the correct selection of the preferable model in both the completely certain and uncertain cases.
READ FULL TEXT