A General Language Assistant as a Laboratory for Alignment

12/01/2021
by   Amanda Askell, et al.
11

Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. As an initial foray in this direction we study simple baseline techniques and evaluations, such as prompting. We find that the benefits from modest interventions increase with model size, generalize to a variety of alignment evaluations, and do not compromise the performance of large models. Next we investigate scaling trends for several training objectives relevant to alignment, comparing imitation learning, binary discrimination, and ranked preference modeling. We find that ranked preference modeling performs much better than imitation learning, and often scales more favorably with model size. In contrast, binary discrimination typically performs and scales very similarly to imitation learning. Finally we study a `preference model pre-training' stage of training, with the goal of improving sample efficiency when finetuning on human preferences.

READ FULL TEXT
07/09/2019

Ranking-Based Reward Extrapolation without Rankings

The performance of imitation learning is typically upper-bounded by the ...
11/05/2020

HILONet: Hierarchical Imitation Learning from Non-Aligned Observations

It is challenging learning from demonstrated observation-only trajectori...
04/12/2022

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

We apply preference modeling and reinforcement learning from human feedb...
12/07/2019

Learning Norms from Stories: A Prior for Value Aligned Agents

Value alignment is a property of an intelligent agent indicating that it...
09/22/2021

Imitation Learning of Stabilizing Policies for Nonlinear Systems

There has been a recent interest in imitation learning methods that are ...
02/17/2021

Fully General Online Imitation Learning

In imitation learning, imitators and demonstrators are policies for pick...
11/15/2021

Scaling Law for Recommendation Models: Towards General-purpose User Representations

A recent trend shows that a general class of models, e.g., BERT, GPT-3, ...