The role of dimensionality reduction in linear classification

by   Weiran Wang, et al.

Dimensionality reduction (DR) is often used as a preprocessing step in classification, but usually one first fixes the DR mapping, possibly using label information, and then learns a classifier (a filter approach). Best performance would be obtained by optimizing the classification error jointly over DR mapping and classifier (a wrapper approach), but this is a difficult nonconvex problem, particularly with nonlinear DR. Using the method of auxiliary coordinates, we give a simple, efficient algorithm to train a combination of nonlinear DR and a classifier, and apply it to a RBF mapping with a linear SVM. This alternates steps where we train the RBF mapping and a linear SVM as usual regression and classification, respectively, with a closed-form step that coordinates both. The resulting nonlinear low-dimensional classifier achieves classification errors competitive with the state-of-the-art but is fast at training and testing, and allows the user to trade off runtime for classification accuracy easily. We then study the role of nonlinear DR in linear classification, and the interplay between the DR mapping, the number of latent dimensions and the number of classes. When trained jointly, the DR mapping takes an extreme role in eliminating variation: it tends to collapse classes in latent space, erasing all manifold structure, and lay out class centroids so they are linearly separable with maximum margin.


page 1

page 2

page 3

page 4


Transformer-based dimensionality reduction

Recently, Transformer is much popular and plays an important role in the...

A more globally accurate dimensionality reduction method using triplets

We first show that the commonly used dimensionality reduction (DR) metho...

Interpretable Discriminative Dimensionality Reduction and Feature Selection on the Manifold

Dimensionality reduction (DR) on the manifold includes effective methods...

On genetic programming representations and fitness functions for interpretable dimensionality reduction

Dimensionality reduction (DR) is an important technique for data explora...

Calibrated Simplex Mapping Classification

We propose a novel supervised multi-class/single-label classifier that m...

An Empirical Study of Dimensional Reduction Techniques for Facial Action Units Detection

Biologically inspired features, such as Gabor filters, result in very hi...

Dimensionality Reduction and Anomaly Detection for CPPS Data using Autoencoder

Unsupervised anomaly detection (AD) is a major topic in the field of Cyb...

Please sign up or login with your details

Forgot password? Click here to reset