What is a Hyperprior?

A hyperprior is an assumption made about a parameter in a prior probability assumption. This is commonly used when the goal is to create conjugate priors, but no specific group of parameters can be inferred from past experiments or subjective analysis. So a probability distribution is used to represent a parameter’s value and even existence in the prior, which is itself a probability distribution covering the parameters of the not-yet-observed data sample.

When are Hyperpriors Used?

While guessing the range of a guess’s parameters by itself is not particularly accurate, this technique allows Bayesian models to be much more dynamic and account for uncertainty, while still using conjugate probabilities to simplify calculations. So instead of using a fixed set of parameters as a conjugate prior, which forces the posterior (after experimentation) probability to only work with that distribution form, the model can add or subtract new parameters and update their values as more data becomes available.

Common Probability Distribution Parameterizations in Machine Learning:

While all probability models follow either Bayesian or Frequentist inference, they can yield vastly different results depending upon what specific parameter distribution algorithm is employed.