Statistical optimality and stability of tangent transform algorithms in logit models

10/25/2020
by   Indrajit Ghosh, et al.
0

A systematic approach to finding variational approximation in an otherwise intractable non-conjugate model is to exploit the general principle of convex duality by minorizing the marginal likelihood that renders the problem tractable. While such approaches are popular in the context of variational inference in non-conjugate Bayesian models, theoretical guarantees on statistical optimality and algorithmic convergence are lacking. Focusing on logistic regression models, we provide mild conditions on the data generating process to derive non-asymptotic upper bounds to the risk incurred by the variational optima. We demonstrate that these assumptions can be completely relaxed if one considers a slight variation of the algorithm by raising the likelihood to a fractional power. Next, we utilize the theory of dynamical systems to provide convergence guarantees for such algorithms in logistic and multinomial logit regression. In particular, we establish local asymptotic stability of the algorithm without any assumptions on the data-generating process. We explore a special case involving a semi-orthogonal design under which a global convergence is obtained. The theory is further illustrated using several numerical studies.

READ FULL TEXT
research
06/01/2023

On the Convergence of Coordinate Ascent Variational Inference

As a computational alternative to Markov chain Monte Carlo approaches, v...
research
10/22/2020

Spike and slab variational Bayes for high dimensional logistic regression

Variational Bayes (VB) is a popular scalable alternative to Markov chain...
research
12/02/2019

Stochastic Variational Inference via Upper Bound

Stochastic variational inference (SVI) plays a key role in Bayesian deep...
research
11/05/2021

Optimality of variational inference for stochastic block model with missing links

Variational methods are extremely popular in the analysis of network dat...
research
02/14/2012

Variational Algorithms for Marginal MAP

Marginal MAP problems are notoriously difficult tasks for graphical mode...
research
01/03/2023

Introducing Variational Inference in Statistics and Data Science Curriculum

Probabilistic models such as logistic regression, Bayesian classificatio...
research
07/06/2022

Gradient-Free Kernel Stein Discrepancy

Stein discrepancies have emerged as a powerful statistical tool, being a...

Please sign up or login with your details

Forgot password? Click here to reset