The Performance Analysis of Generalized Margin Maximizer (GMM) on Separable Data

by   Fariborz Salehi, et al.

Logistic models are commonly used for binary classification tasks. The success of such models has often been attributed to their connection to maximum-likelihood estimators. It has been shown that gradient descent algorithm, when applied on the logistic loss, converges to the max-margin classifier (a.k.a. hard-margin SVM). The performance of the max-margin classifier has been recently analyzed. Inspired by these results, in this paper, we present and study a more general setting, where the underlying parameters of the logistic model possess certain structures (sparse, block-sparse, low-rank, etc.) and introduce a more general framework (which is referred to as "Generalized Margin Maximizer", GMM). While classical max-margin classifiers minimize the 2-norm of the parameter vector subject to linearly separating the data, GMM minimizes any arbitrary convex function of the parameter vector. We provide a precise analysis of the performance of GMM via the solution of a system of nonlinear equations. We also provide a detailed study for three special cases: (1) ℓ_2-GMM that is the max-margin classifier, (2) ℓ_1-GMM which encourages sparsity, and (3) ℓ_∞-GMM which is often used when the parameter vector has binary entries. Our theoretical results are validated by extensive simulation results across a range of parameter values, problem instances, and model structures.



There are no comments yet.


page 4


The Impact of Regularization on High-dimensional Logistic Regression

Logistic regression is commonly used for modeling dichotomous outcomes. ...

A Model of Double Descent for High-dimensional Binary Linear Classification

We consider a model for logistic regression where only a subset of featu...

Inductive Bias of Gradient Descent based Adversarial Training on Separable Data

Adversarial training is a principled approach for training robust neural...

MedLDA: A General Framework of Maximum Margin Supervised Topic Models

Supervised topic models utilize document's side information for discover...

Implicitly Maximizing Margins with the Hinge Loss

A new loss function is proposed for neural networks on classification ta...

Max-Margin is Dead, Long Live Max-Margin!

The foundational concept of Max-Margin in machine learning is ill-posed ...

Non-Gaussian Discriminative Factor Models via the Max-Margin Rank-Likelihood

We consider the problem of discriminative factor analysis for data that ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.