Improving Regression Performance with Distributional Losses

06/12/2018
by   Ehsan Imani, et al.
0

There is growing evidence that converting targets to soft targets in supervised learning can provide considerable gains in performance. Much of this work has considered classification, converting hard zero-one values to soft labels---such as by adding label noise, incorporating label ambiguity or using distillation. In parallel, there is some evidence from a regression setting in reinforcement learning that learning distributions can improve performance. In this work, we investigate the reasons for this improvement, in a regression setting. We introduce a novel distributional regression loss, and similarly find it significantly improves prediction accuracy. We investigate several common hypotheses, around reducing overfitting and improved representations. We instead find evidence for an alternative hypothesis: this loss is easier to optimize, with better behaved gradients, resulting in improved generalization. We provide theoretical support for this alternative hypothesis, by characterizing the norm of the gradients of this loss.

READ FULL TEXT
research
11/11/2022

Continuous Soft Pseudo-Labeling in ASR

Continuous pseudo-labeling (PL) algorithms such as slimIPL have recently...
research
10/01/2021

A Cramér Distance perspective on Non-crossing Quantile Regression in Distributional Reinforcement Learning

Distributional reinforcement learning (DRL) extends the value-based appr...
research
07/23/2021

Similarity Based Label Smoothing For Dialogue Generation

Generative neural conversational systems are generally trained with the ...
research
06/15/2023

Partial-Label Regression

Partial-label learning is a popular weakly supervised learning setting t...
research
09/20/2020

Learning Soft Labels via Meta Learning

One-hot labels do not represent soft decision boundaries among concepts,...
research
08/24/2021

Improving Object Detection by Label Assignment Distillation

Label assignment in object detection aims to assign targets, foreground ...
research
12/16/2021

Knowledge Distillation Leveraging Alternative Soft Targets from Non-Parallel Qualified Speech Data

This paper describes a novel knowledge distillation framework that lever...

Please sign up or login with your details

Forgot password? Click here to reset