DeepAI AI Chat
Log In Sign Up

Boosted Prompt Ensembles for Large Language Models

by   Silviu Pitis, et al.

Methods such as chain-of-thought prompting and self-consistency have pushed the frontier of language model reasoning performance with no additional training. To further improve performance, we propose a prompt ensembling method for large language models, which uses a small dataset to construct a set of few shot prompts that together comprise a “boosted prompt ensemble”. The few shot examples for each prompt are chosen in a stepwise fashion to be “hard” examples on which the previous step's ensemble is uncertain. We show that this outperforms single-prompt output-space ensembles and bagged prompt-space ensembles on the GSM8k and AQuA datasets, among others. We propose both train-time and test-time versions of boosted prompting that use different levels of available annotation and conduct a detailed empirical study of our algorithm.


page 1

page 2

page 3

page 4


Self-Consistency Improves Chain of Thought Reasoning in Language Models

We explore a simple ensemble strategy, self-consistency, that significan...

Rationale-Augmented Ensembles in Language Models

Recent research has shown that rationales, or step-by-step chains of tho...

The Infinitesimal Jackknife and Combinations of Models

The Infinitesimal Jackknife is a general method for estimating variances...

Embroid: Unsupervised Prediction Smoothing Can Improve Few-Shot Classification

Recent work has shown that language models' (LMs) prompt-based learning ...

Large Language Models are reasoners with Self-Verification

When a large language model (LLM) performs complex reasoning by chain of...

Short-answer scoring with ensembles of pretrained language models

We investigate the effectiveness of ensembles of pretrained transformer-...

Forming Ensembles at Runtime: A Machine Learning Approach

Smart system applications (SSAs) built on top of cyber-physical and soci...