DeepAI AI Chat
Log In Sign Up

Evaluating Creative Language Generation: The Case of Rap Lyric Ghostwriting

by   Peter Potash, et al.
UMass Lowell

Language generation tasks that seek to mimic human ability to use language creatively are difficult to evaluate, since one must consider creativity, style, and other non-trivial aspects of the generated text. The goal of this paper is to develop evaluation methods for one such task, ghostwriting of rap lyrics, and to provide an explicit, quantifiable foundation for the goals and future directions of this task. Ghostwriting must produce text that is similar in style to the emulated artist, yet distinct in content. We develop a novel evaluation methodology that addresses several complementary aspects of this task, and illustrate how such evaluation can be used to meaningfully analyze system performance. We provide a corpus of lyrics for 13 rap artists, annotated for stylistic similarity, which allows us to assess the feasibility of manual evaluation for generated verse.


page 1

page 2

page 3

page 4


Introducing Aspects of Creativity in Automatic Poetry Generation

Poetry Generation involves teaching systems to automatically generate te...

Controlling Linguistic Style Aspects in Neural Language Generation

Most work on neural natural language generation (NNLG) focus on controll...

A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems

Most Natural Language Generation systems need to produce accurate texts....

Sentence-Level Content Planning and Style Specification for Neural Text Generation

Building effective text generation systems requires three critical compo...

Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework

Style is an integral part of natural language. However, evaluation metho...

Dynamic Human Evaluation for Relative Model Comparisons

Collecting human judgements is currently the most reliable evaluation me...

The Validity, Generalizability and Feasibility of Summative Evaluation Methods in Visual Analytics

Many evaluation methods have been used to assess the usefulness of Visua...