Weight Vector Tuning and Asymptotic Analysis of Binary Linear Classifiers

by   Lama B. Niyazi, et al.

Unlike its intercept, a linear classifier's weight vector cannot be tuned by a simple grid search. Hence, this paper proposes weight vector tuning of a generic binary linear classifier through the parameterization of a decomposition of the discriminant by a scalar which controls the trade-off between conflicting informative and noisy terms. By varying this parameter, the original weight vector is modified in a meaningful way. Applying this method to a number of linear classifiers under a variety of data dimensionality and sample size settings reveals that the classification performance loss due to non-optimal native hyperparameters can be compensated for by weight vector tuning. This yields computational savings as the proposed tuning method reduces to tuning a scalar compared to tuning the native hyperparameter, which may involve repeated weight vector generation along with its burden of optimization, dimensionality reduction, etc., depending on the classifier. It is also found that weight vector tuning significantly improves the performance of Linear Discriminant Analysis (LDA) under high estimation noise. Proceeding from this second finding, an asymptotic study of the misclassification probability of the parameterized LDA classifier in the growth regime where the data dimensionality and sample size are comparable is conducted. Using random matrix theory, the misclassification probability is shown to converge to a quantity that is a function of the true statistics of the data. Additionally, an estimator of the misclassification probability is derived. Finally, computationally efficient tuning of the parameter using this estimator is demonstrated on real data.



There are no comments yet.


page 1

page 2

page 3

page 4


Asymptotic Analysis of an Ensemble of Randomly Projected Linear Discriminants

Datasets from the fields of bioinformatics, chemometrics, and face recog...

A Doubly Regularized Linear Discriminant Analysis Classifier with Automatic Parameter Selection

Linear discriminant analysis (LDA) based classifiers tend to falter in m...

Nested Cavity Classifier: performance and remedy

Nested Cavity Classifier (NCC) is a classification rule that pursues par...

Linear Classifiers Under Infinite Imbalance

We study the behavior of linear discriminant functions for binary classi...

Revisiting Classical Multiclass Linear Discriminant Analysis with a Novel Prototype-based Interpretable Solution

Linear discriminant analysis (LDA) is a fundamental method for feature e...

Classification with imperfect training labels

We study the effect of imperfect training data labels on the performance...

On the Properties of Error Patterns in the Constant Lee Weight Channel

The problem of scalar multiplication applied to vectors is considered in...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.