Similarity Based Label Smoothing For Dialogue Generation

07/23/2021
by   Sougata Saha, et al.
0

Generative neural conversational systems are generally trained with the objective of minimizing the entropy loss between the training "hard" targets and the predicted logits. Often, performance gains and improved generalization can be achieved by using regularization techniques like label smoothing, which converts the training "hard" targets to "soft" targets. However, label smoothing enforces a data independent uniform distribution on the incorrect training targets, which leads to an incorrect assumption of equi-probable incorrect targets for each correct target. In this paper we propose and experiment with incorporating data dependent word similarity based weighing methods to transforms the uniform distribution of the incorrect target probabilities in label smoothing, to a more natural distribution based on semantics. We introduce hyperparameters to control the incorrect target distribution, and report significant performance gains over networks trained using standard label smoothing based loss, on two standard open domain dialogue corpora.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/06/2019

When Does Label Smoothing Help?

The generalization and learning speed of a multi-class neural network ca...
research
05/30/2021

Diversifying Dialog Generation via Adaptive Label Smoothing

Neural dialogue generation models trained with the one-hot target distri...
research
09/14/2020

Adaptive Label Smoothing

This paper concerns the use of objectness measures to improve the calibr...
research
06/12/2018

Improving Regression Performance with Distributional Losses

There is growing evidence that converting targets to soft targets in sup...
research
05/02/2020

Generalized Entropy Regularization or: There's Nothing Special about Label Smoothing

Prior work has explored directly regularizing the output distributions o...
research
12/02/2020

Regularization via Adaptive Pairwise Label Smoothing

Label Smoothing (LS) is an effective regularizer to improve the generali...
research
05/15/2023

Label Smoothing is Robustification against Model Misspecification

Label smoothing (LS) adopts smoothed targets in classification tasks. Fo...

Please sign up or login with your details

Forgot password? Click here to reset