Low Complexity Approximate Bayesian Logistic Regression for Sparse Online Learning

by   Gil I. Shamir, et al.

Theoretical results show that Bayesian methods can achieve lower bounds on regret for online logistic regression. In practice, however, such techniques may not be feasible especially for very large feature sets. Various approximations that, for huge sparse feature sets, diminish the theoretical advantages, must be used. Often, they apply stochastic gradient methods with hyper-parameters that must be tuned on some surrogate loss, defeating theoretical advantages of Bayesian methods. The surrogate loss, defined to approximate the mixture, requires techniques as Monte Carlo sampling, increasing computations per example. We propose low complexity analytical approximations for sparse online logistic and probit regressions. Unlike variational inference and other methods, our methods use analytical closed forms, substantially lowering computations. Unlike dense solutions, as Gaussian Mixtures, our methods allow for sparse problems with huge feature sets without increasing complexity. With the analytical closed forms, there is also no need for applying stochastic gradient methods on surrogate losses, and for tuning and balancing learning and regularization hyper-parameters. Empirical results top the performance of the more computationally involved methods. Like such methods, our methods still reveal per feature and per example uncertainty measures.


page 1

page 2

page 3

page 4


Stochastic Variational Inference via Upper Bound

Stochastic variational inference (SVI) plays a key role in Bayesian deep...

Variational Inference in Nonconjugate Models

Mean-field variational methods are widely used for approximate posterior...

PASS-GLM: polynomial approximate sufficient statistics for scalable Bayesian GLM inference

Generalized linear models (GLMs) -- such as logistic regression, Poisson...

Bounding Wasserstein distance with couplings

Markov chain Monte Carlo (MCMC) provides asymptotically consistent estim...

Fast Sparse Classification for Generalized Linear and Additive Models

We present fast classification techniques for sparse generalized linear ...

Belief Flows of Robust Online Learning

This paper introduces a new probabilistic model for online learning whic...

Please sign up or login with your details

Forgot password? Click here to reset