From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification

02/05/2016
by   André F. T. Martins, et al.
0

We propose sparsemax, a new activation function similar to the traditional softmax, but able to output sparse probabilities. After deriving its properties, we show how its Jacobian can be efficiently computed, enabling its use in a network trained with backpropagation. Then, we propose a new smooth and convex loss function which is the sparsemax analogue of the logistic loss. We reveal an unexpected connection between this new loss and the Huber classification loss. We obtain promising empirical results in multi-label classification problems and in attention-based neural networks for natural language inference. For the latter, we achieve a similar performance as the traditional softmax, but with a selective, more compact, attention focus.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/23/2021

Sparse-softmax: A Simpler and Faster Alternative Softmax Transformation

The softmax function is widely used in artificial neural networks for th...
research
03/31/2023

A two-head loss function for deep Average-K classification

Average-K classification is an alternative to top-K classification in wh...
research
02/03/2018

Joint Binary Neural Network for Multi-label Learning with Applications to Emotion Classification

Recently the deep learning techniques have achieved success in multi-lab...
research
04/11/2023

r-softmax: Generalized Softmax with Controllable Sparsity Rate

Nowadays artificial neural network models achieve remarkable results in ...
research
03/14/2017

On the benefits of output sparsity for multi-label classification

The multi-label classification framework, where each observation can be ...
research
05/02/2022

Jam or Cream First? Modeling Ambiguity in Neural Machine Translation with SCONES

The softmax layer in neural machine translation is designed to model the...
research
01/28/2019

Additive Margin SincNet for Speaker Recognition

Speaker Recognition is a challenging task with essential applications su...

Please sign up or login with your details

Forgot password? Click here to reset