DeepAI AI Chat
Log In Sign Up

Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation

by   Xingdi Yuan, et al.

Large Language Models (LLMs) have in recent years demonstrated impressive prowess in natural language generation. A common practice to improve generation diversity is to sample multiple outputs from the model. However, there lacks a simple and robust way of selecting the best output from these stochastic samples. As a case study framed in the context of question generation, we propose two prompt-based approaches to selecting high-quality questions from a set of LLM-generated candidates. Our method works under the constraints of 1) a black-box (non-modifiable) question generation model and 2) lack of access to human-annotated references – both of which are realistic limitations for real-world deployment of LLMs. With automatic as well as human evaluations, we empirically demonstrate that our approach can effectively select questions of higher qualities than greedy generation.


"What makes a question inquisitive?" A Study on Type-Controlled Inquisitive Question Generation

We propose a type-controlled framework for inquisitive question generati...

Can Very Large Pretrained Language Models Learn Storytelling With A Few Examples?

While pre-trained language models can generate individually fluent sente...

Investigating the use of Paraphrase Generation for Question Reformulation in the FRANK QA system

We present a study into the ability of paraphrase generation methods to ...

QRelScore: Better Evaluating Generated Questions with Deeper Understanding of Context-aware Relevance

Existing metrics for assessing question generation not only require cost...

On the Evaluation of Answer-Agnostic Paragraph-level Multi-Question Generation

We study the task of predicting a set of salient questions from a given ...

Automatic Sampling of Geographic objects

Today, one's disposes of large datasets composed of thousands of geograp...

Math Word Problem Generation with Mathematical Consistency and Problem Context Constraints

We study the problem of generating arithmetic math word problems (MWPs) ...