Reconsidering the Past: Optimizing Hidden States in Language Models

12/16/2021
by   Davis Yoshida, et al.
0

We present Hidden-State Optimization (HSO), a gradient-based method for improving the performance of transformer language models at inference time. Similar to dynamic evaluation (Krause et al., 2018), HSO computes the gradient of the log-probability the language model assigns to an evaluation text, but uses it to update the cached hidden states rather than the model parameters. We test HSO with pretrained Transformer-XL and GPT-2 language models, finding improvement on the WikiText103 and PG-19 datasets in terms of perplexity, especially when evaluating a model outside of its training distribution. We also demonstrate downstream applicability by showing gains in the recently developed prompt-based few-shot evaluation setting, again with no extra parameters or training data.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/05/2022

Meta-Learning Fast Weight Language Models

Dynamic evaluation of language models (LMs) adapts model parameters at t...
research
05/24/2022

Garden-Path Traversal within GPT-2

In recent years, massive language models consisting exclusively of trans...
research
03/29/2022

Training Compute-Optimal Large Language Models

We investigate the optimal model size and number of tokens for training ...
research
02/13/2023

Gradient-Based Automated Iterative Recovery for Parameter-Efficient Tuning

Pretrained large language models (LLMs) are able to solve a wide variety...
research
04/07/2020

Byte Pair Encoding is Suboptimal for Language Model Pretraining

The success of pretrained transformer language models in natural languag...
research
08/17/2023

PMET: Precise Model Editing in a Transformer

Model editing techniques modify a minor proportion of knowledge in Large...
research
12/20/2022

Language Modeling with Latent Situations

Language models (LMs) often generate incoherent outputs: they refer to e...

Please sign up or login with your details

Forgot password? Click here to reset