Mixture weights optimisation for Alpha-Divergence Variational Inference

06/09/2021
by   Kamélia Daudel, et al.
0

This paper focuses on α-divergence minimisation methods for Variational Inference. More precisely, we are interested in algorithms optimising the mixture weights of any given mixture model, without any information on the underlying distribution of its mixture components parameters. The Power Descent, defined for all α≠ 1, is one such algorithm and we establish in our work the full proof of its convergence towards the optimal mixture weights when α <1. Since the α-divergence recovers the widely-used forward Kullback-Leibler when α→ 1, we then extend the Power Descent to the case α = 1 and show that we obtain an Entropic Mirror Descent. This leads us to investigate the link between Power Descent and Entropic Mirror Descent: first-order approximations allow us to introduce the Renyi Descent, a novel algorithm for which we prove an O(1/N) convergence rate. Lastly, we compare numerically the behavior of the unbiased Power Descent and of the biased Renyi Descent and we discuss the potential advantages of one algorithm over the other.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/20/2020

Infinite-dimensional gradient-based descent for alpha-divergence minimisation

This paper introduces the (α, Γ)-descent, an iterative algorithm which o...
research
06/27/2023

On fixed and uncertain mixture prior weights

This paper focuses on the specification of the weights for the component...
research
03/09/2021

Monotonic Alpha-divergence Minimisation

In this paper, we introduce a novel iterative algorithm which carries ou...
research
05/23/2019

Variational Inference with Mixture Model Approximation: Robotic Applications

We propose a method to approximate the distribution of robot configurati...
research
07/05/2021

The Last-Iterate Convergence Rate of Optimistic Mirror Descent in Stochastic Variational Inequalities

In this paper, we analyze the local convergence rate of optimistic mirro...
research
10/12/2022

Alpha-divergence Variational Inference Meets Importance Weighted Auto-Encoders: Methodology and Asymptotics

Several algorithms involving the Variational Rényi (VR) bound have been ...
research
10/07/2020

Learning from demonstration using products of experts: applications to manipulation and task prioritization

Probability distributions are key components of many learning from demon...

Please sign up or login with your details

Forgot password? Click here to reset