Measuring Massive Multitask Language Understanding

09/07/2020
by   Dan Hendrycks, et al.
28

We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.

READ FULL TEXT
research
04/25/2023

Measuring Massive Multitask Chinese Understanding

The development of large-scale Chinese language models is flourishing, y...
research
06/15/2023

CMMLU: Measuring massive multitask language understanding in Chinese

As the capabilities of large language models (LLMs) continue to advance,...
research
03/04/2020

jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models

We introduce jiant, an open source toolkit for conducting multitask and ...
research
05/06/2020

Multitask Models for Supervised Protests Detection in Texts

The CLEF 2019 ProtestNews Lab tasks participants to identify text relati...
research
03/24/2021

Active Multitask Learning with Committees

The cost of annotating training data has traditionally been a bottleneck...
research
05/24/2023

Estimating Large Language Model Capabilities without Labeled Test Data

Large Language Models (LLMs) have exhibited an impressive ability to per...

Please sign up or login with your details

Forgot password? Click here to reset