Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints

02/17/2023
by   Albert Lu, et al.
0

The limits of open-ended generative models are unclear, yet increasingly important. What causes them to succeed and what causes them to fail? In this paper, we take a prompt-centric approach to analyzing and bounding the abilities of open-ended generative models. We present a generic methodology of analysis with two challenging prompt constraint types: structural and stylistic. These constraint types are categorized into a set of well-defined constraints that are analyzable by a single prompt. We then systematically create a diverse set of simple, natural, and useful prompts to robustly analyze each individual constraint. Using the GPT-3 text-davinci-002 model as a case study, we generate outputs from our collection of prompts and analyze the model's generative failures. We also show the generalizability of our proposed method on other large models like BLOOM and OPT. Our results and our in-context mitigation strategies reveal open challenges for future research. We have publicly released our code at https://github.com/SALT-NLP/Bound-Cap-LLM.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/17/2023

COLLIE: Systematic Construction of Constrained Text Generation Tasks

Text generation under constraints have seen increasing interests in natu...
research
07/27/2023

Evaluating Generative Models for Graph-to-Text Generation

Large language models (LLMs) have been widely employed for graph-to-text...
research
03/19/2021

Controllable Generation from Pre-trained Language Models via Inverse Prompting

Large-scale pre-trained language models have demonstrated strong capabil...
research
03/22/2023

MEGA: Multilingual Evaluation of Generative AI

Generative AI models have impressive performance on many Natural Languag...
research
12/20/2022

Controllable Text Generation with Language Constraints

We consider the task of text generation in language models with constrai...
research
08/18/2023

A Methodology for Generative Spelling Correction via Natural Spelling Errors Emulation across Multiple Domains and Languages

Modern large language models demonstrate impressive capabilities in text...
research
03/08/2023

disco: a toolkit for Distributional Control of Generative Models

Pre-trained language models and other generative models have revolutioni...

Please sign up or login with your details

Forgot password? Click here to reset