Discovering Latent Knowledge in Language Models Without Supervision

by   Collin Burns, et al.

Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4% on average. We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don't have access to explicit ground truth labels.


page 7

page 18

page 19


Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering

Large Language Models (LLMs) are capable of performing zero-shot closed-...

CLAM: Selective Clarification for Ambiguous Questions with Large Language Models

State-of-the-art language models are often accurate on many question-ans...

How Context Affects Language Models' Factual Predictions

When pre-trained on large unsupervised textual corpora, language models ...

How Can We Know What Language Models Know?

Recent work has presented intriguing results examining the knowledge con...

Navigating Human Language Models with Synthetic Agents

Modern natural language models such as the GPT-2/GPT-3 contain tremendou...

LM vs LM: Detecting Factual Errors via Cross Examination

A prominent weakness of modern language models (LMs) is their tendency t...

Despite "super-human" performance, current LLMs are unsuited for decisions about ethics and safety

Large language models (LLMs) have exploded in popularity in the past few...

Please sign up or login with your details

Forgot password? Click here to reset