DeepAI AI Chat
Log In Sign Up

Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation

09/22/2022
by   Xingdi Yuan, et al.
Microsoft
2

Large Language Models (LLMs) have in recent years demonstrated impressive prowess in natural language generation. A common practice to improve generation diversity is to sample multiple outputs from the model. However, there lacks a simple and robust way of selecting the best output from these stochastic samples. As a case study framed in the context of question generation, we propose two prompt-based approaches to selecting high-quality questions from a set of LLM-generated candidates. Our method works under the constraints of 1) a black-box (non-modifiable) question generation model and 2) lack of access to human-annotated references – both of which are realistic limitations for real-world deployment of LLMs. With automatic as well as human evaluations, we empirically demonstrate that our approach can effectively select questions of higher qualities than greedy generation.

READ FULL TEXT
05/17/2022

"What makes a question inquisitive?" A Study on Type-Controlled Inquisitive Question Generation

We propose a type-controlled framework for inquisitive question generati...
01/24/2023

Can Very Large Pretrained Language Models Learn Storytelling With A Few Examples?

While pre-trained language models can generate individually fluent sente...
06/06/2022

Investigating the use of Paraphrase Generation for Question Reformulation in the FRANK QA system

We present a study into the ability of paraphrase generation methods to ...
04/29/2022

QRelScore: Better Evaluating Generated Questions with Deeper Understanding of Context-aware Relevance

Existing metrics for assessing question generation not only require cost...
03/09/2022

On the Evaluation of Answer-Agnostic Paragraph-level Multi-Question Generation

We study the task of predicting a set of salient questions from a given ...
04/20/2012

Automatic Sampling of Geographic objects

Today, one's disposes of large datasets composed of thousands of geograp...
09/09/2021

Math Word Problem Generation with Mathematical Consistency and Problem Context Constraints

We study the problem of generating arithmetic math word problems (MWPs) ...