A Variational Formula for Rényi Divergences
We derive a new variational formula for the Rényi family of divergences, R_α(QP), generalizing the classical Donsker-Varadhan variational formula for the Kullback-Leibler divergence. The objective functional in this new variational representation is expressed in terms of expectations under Q and P, and hence can be estimated using samples from the two distributions. We illustrate the utility of such a variational formula by constructing neural-network estimators for the Rényi divergences.
READ FULL TEXT