Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation

by   Cristina Garbacea, et al.

Recent advances in deep learning have resulted in a resurgence in the popularity of natural language generation (NLG). Many deep learning based models, including recurrent neural networks and generative adversarial networks, have been proposed and applied to generating various types of text. Despite the fast development of methods, how to better evaluate the quality of these natural language generators remains a significant challenge. We conduct an in-depth empirical study to evaluate the existing evaluation methods for natural language generation. We compare human-based evaluators with a variety of automated evaluation procedures, including discriminative evaluators that measure how well the generated text can be distinguished from human-written text, as well as text overlap metrics that measure how similar the generated text is to human-written references. We measure to what extent these different evaluators agree on the ranking of a dozen of state-of-the-art generators for online product reviews. We find that human evaluators do not correlate well with discriminative evaluators, leaving a bigger question of whether adversarial accuracy is the correct objective for natural language generation. In general, distinguishing machine-generated text is a challenging task even for human evaluators, and their decisions tend to correlate better with text overlap metrics. We also find that diversity is an intriguing metric that is indicative of the assessments of different evaluators.


page 1

page 2

page 3

page 4


Why is constrained neural language generation particularly challenging?

Recent advances in deep neural language models combined with the capacit...

Russian Natural Language Generation: Creation of a Language Modelling Dataset and Evaluation with Modern Neural Architectures

Generating coherent, grammatically correct, and meaningful text is very ...

Automating Text Naturalness Evaluation of NLG Systems

Automatic methods and metrics that assess various quality criteria of au...

Perceived Trustworthiness of Natural Language Generators

Natural Language Generation tools, such as chatbots that can generate hu...

A Hybrid Natural Language Generation System Integrating Rules and Deep Learning Algorithms

This paper proposes an enhanced natural language generation system combi...

Learning to Write with Cooperative Discriminators

Recurrent Neural Networks (RNNs) are powerful autoregressive sequence mo...

Creative Data Generation: A Review Focusing on Text and Poetry

The rapid advancement in machine learning has led to a surge in automati...

Please sign up or login with your details

Forgot password? Click here to reset