Maximum Class Separation as Inductive Bias in One Matrix

06/17/2022
by   Tejaswi Kasarla, et al.
11

Maximizing the separation between classes constitutes a well-known inductive bias in machine learning and a pillar of many traditional algorithms. By default, deep networks are not equipped with this inductive bias and therefore many alternative solutions have been proposed through differential optimization. Current approaches tend to optimize classification and separation jointly: aligning inputs with class vectors and separating class vectors angularly. This paper proposes a simple alternative: encoding maximum separation as an inductive bias in the network by adding one fixed matrix multiplication before computing the softmax activations. The main observation behind our approach is that separation does not require optimization but can be solved in closed-form prior to training and plugged into a network. We outline a recursive approach to obtain the matrix consisting of maximally separable vectors for any number of classes, which can be added with negligible engineering effort and computational overhead. Despite its simple nature, this one matrix multiplication provides real impact. We show that our proposal directly boosts classification, long-tailed recognition, out-of-distribution detection, and open-set recognition, from CIFAR to ImageNet. We find empirically that maximum separation works best as a fixed bias; making the matrix learnable adds nothing to the performance. The closed-form implementation and code to reproduce the experiments are on github.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/23/2023

A simple division-free algorithm for computing Pfaffians

We present a very simple algorithm for computing Pfaffians which uses no...
research
06/17/2020

Neural Anisotropy Directions

In this work, we analyze the role of the network architecture in shaping...
research
02/22/2023

Matrix Multiplication and Number On the Forehead Communication

Three-player Number On the Forehead communication may be thought of as a...
research
05/12/2023

AMULET: Adaptive Matrix-Multiplication-Like Tasks

Many useful tasks in data science and machine learning applications can ...
research
06/16/2016

Estimation of matrix trace using machine learning

We present a new trace estimator of the matrix whose explicit form is no...

Please sign up or login with your details

Forgot password? Click here to reset