Style Transfer for Texts: to Err is Human, but Error Margins Matter

08/19/2019
by   Alexey Tikhonov, et al.
0

This paper shows that standard assessment methodology for style transfer has several significant problems. First, the standard metrics for style accuracy and semantics preservation vary significantly on different re-runs. Therefore one has to report error margins for the obtained results. Second, starting with certain values of bilingual evaluation understudy (BLEU) between input and output and accuracy of the sentiment transfer the optimization of these two standard metrics diverge from the intuitive goal of the style transfer task. Finally, due to the nature of the task itself, there is a specific dependence between these two metrics that could be easily manipulated. Under these circumstances, we suggest taking BLEU between input and human-written reformulations into consideration for benchmarks. We also propose three new architectures that outperform state of the art in terms of this metric.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/19/2019

Style Transfer for Texts: Retrain, Report Errors, Compare with Rewrites

This paper shows that standard assessment methodology for style transfer...
research
09/26/2019

Decomposing Textual Information For Style Transfer

This paper focuses on latent representations that could effectively deco...
research
01/05/2021

On the interaction of automatic evaluation and task framing in headline style transfer

An ongoing debate in the NLG community concerns the best way to evaluate...
research
04/10/2020

Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metric

The rapid development of such natural language processing tasks as style...
research
04/04/2019

Evaluating Style Transfer for Text

Research in the area of style transfer for text is currently bottlenecke...
research
08/13/2018

What is wrong with style transfer for texts?

A number of recent machine learning papers work with an automated style ...
research
04/15/2022

Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfer

Although text style transfer has witnessed rapid development in recent y...

Please sign up or login with your details

Forgot password? Click here to reset