What Makes Data-to-Text Generation Hard for Pretrained Language Models?

05/23/2022
by   Moniba Keymanesh, et al.
0

Expressing natural language descriptions of structured facts or relations – data-to-text generation (D2T) – increases the accessibility of structured knowledge repositories. Previous work shows that pre-trained language models(PLMs) perform remarkably well on this task after fine-tuning on a significant amount of task-specific training data. On the other hand, while auto-regressive PLMs can generalize from a few task examples, their efficacy at D2T is largely unexplored. Furthermore, we have an incomplete understanding of the limits of PLMs on D2T. In this work, we conduct an empirical study of both fine-tuned and auto-regressive PLMs on the DART multi-domain D2T dataset. We consider their performance as a function of the amount of task-specific data and how these data are incorporated into the models: zero and few-shot learning, and fine-tuning of model weights. In addition, we probe the limits of PLMs by measuring performance on subsets of the evaluation data: novel predicates and abstractive test examples. To improve the performance on these subsets, we investigate two techniques: providing predicate descriptions in the context and re-ranking generated candidates by information reflected in the source. Finally, we conduct a human evaluation of model errors and show that D2T generation tasks would benefit from datasets with more careful manual curation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/09/2022

Generating Training Data with Language Models: Towards Zero-Shot Language Understanding

Pretrained language models (PLMs) have demonstrated remarkable performan...
research
12/08/2020

Facts2Story: Controlling Text Generation by Key Facts

Recent advancements in self-attention neural network architectures have ...
research
03/28/2023

On Codex Prompt Engineering for OCL Generation: An Empirical Study

The Object Constraint Language (OCL) is a declarative language that adds...
research
06/06/2023

TwistList: Resources and Baselines for Tongue Twister Generation

Previous work in phonetically-grounded language generation has mainly fo...
research
02/08/2023

Auto-Learning: An Adversarial Process of Two Pre-trained Models for Natural Language Generation

Pre-trained models have been used in many fields in recent years, rangin...
research
05/22/2023

Small Language Models Improve Giants by Rewriting Their Outputs

Large language models (LLMs) have demonstrated impressive few-shot learn...
research
12/27/2021

What do Large Language Models Learn about Scripts?

Script Knowledge (Schank and Abelson, 1975) has long been recognized as ...

Please sign up or login with your details

Forgot password? Click here to reset