# Logistic Regression: The Importance of Being Improper

Learning linear predictors with the logistic loss---both in stochastic and online settings---is a fundamental task in learning and statistics, with direct connections to classification and boosting. Existing "fast rates" for this setting exhibit exponential dependence on the predictor norm, and Hazan et al. (2014) showed that this is unfortunately unimprovable. Starting with the simple observation that the logistic loss is 1-mixable, we design a new efficient improper learning algorithm for online logistic regression that circumvents the aforementioned lower bound with a regret bound exhibiting a doubly-exponential improvement in dependence on the predictor norm. This provides a positive resolution to a variant of the COLT 2012 open problem of McMahan and Streeter (2012) when improper learning is allowed. This improvement is obtained both in the online setting and, with some extra work, in the batch statistical setting with high probability. We also show that the improved dependency on predictor norm is also near-optimal. Leveraging this improved dependency on the predictor norm yields the following applications: (a) we give algorithms for online bandit multiclass learning with the logistic loss with an Õ(√(n)) relative mistake bound across essentially all parameter ranges, thus providing a solution to the COLT 2009 open problem of Abernethy and Rakhlin (2009), and (b) we give an adaptive algorithm for online multiclass boosting with optimal sample complexity, thus partially resolving an open problem of Beygelzimer et al. (2015) and Jung et al. (2017). Finally, we give information-theoretic bounds on the optimal rates for improper logistic regression with general function classes, thereby characterizing the extent to which our improvement for linear classes extends to other parameteric and even nonparametric settings.

## Authors

• 26 publications
• 16 publications
• 32 publications
• 36 publications
• 33 publications
• ### Efficient improper learning for online logistic regression

We consider the setting of online logistic regression and consider the r...
03/18/2020 ∙ by Rémi Jézéquel, et al. ∙ 0

• ### An improper estimator with optimal excess risk in misspecified density estimation and logistic regression

We introduce a procedure for predictive conditional density estimation u...

• ### Improved Confidence Bounds for the Linear Logistic Model and Applications to Linear Bandits

We propose improved fixed-design confidence bounds for the linear logist...
11/23/2020 ∙ by Kwang-Sung Jun, et al. ∙ 17

• ### Bias no more: high-probability data-dependent regret bounds for adversarial bandits and MDPs

We develop a new approach to obtaining high probability regret bounds fo...
06/14/2020 ∙ by Chung-Wei Lee, et al. ∙ 0

• ### Efficient Online Bandit Multiclass Learning with Õ(√(T)) Regret

We present an efficient second-order algorithm with Õ(1/η√(T)) regret fo...
02/25/2017 ∙ by Alina Beygelzimer, et al. ∙ 0

• ### Logistic Regression Regret: What's the Catch?

We address the problem of the achievable regret rates with online logist...
02/07/2020 ∙ by Gil I. Shamir, et al. ∙ 0