What are Energy-Based Models in Machine Learning?
Energy-Based Models (EBMs) discover data dependencies by applying a measure of compatibility (scalar energy) to each configuration of the variables. For a model to make a prediction or decision (inference) it needs to set the value of observed variables to 1 and finding values of the remaining variables that minimize that “energy” level.
In the same way, machine learning consists of discovering an energy function that assigns low energies to the correct values of the remaining variables, and higher energies to the incorrect values. A so-called “loss functional,” that’s minimized during training, is used to measure the quality of the energy functions. Within this framework, there are many energy functions and loss functionals allows available to design different probabilistic and non-probabilistic statistical models.
What’s the Advantage of Energy-based Learning?
Energy-based learning is a unified framework for all these probabilistic and non-probabilistic approaches. Especially for non-probabilistic training of graphical or other structured models. This learning approach is an alternative to probabilistic estimation for classification, prediction and decision-making.
Since proper normalization is not required, energy-based approaches don’t have the issues arising from estimating the normalization constant in probabilistic models. Also, the absence of any normalization condition allows for much more flexibility in the design of learning machines.
