What Do NLP Researchers Believe? Results of the NLP Community Metasurvey

by   Julian Michael, et al.
NYU college

We present the results of the NLP Community Metasurvey. Run from May to June 2022, the survey elicited opinions on controversial issues, including industry influence in the field, concerns about AGI, and ethics. Our results put concrete numbers to several controversies: For example, respondents are split almost exactly in half on questions about the importance of artificial general intelligence, whether language models understand language, and the necessity of linguistic structure and inductive bias for solving NLP problems. In addition, the survey posed meta-questions, asking respondents to predict the distribution of survey responses. This allows us not only to gain insight on the spectrum of beliefs held by NLP researchers, but also to uncover false sociological beliefs where the community's predictions don't match reality. We find such mismatches on a wide range of issues. Among other results, the community greatly overestimates its own belief in the usefulness of benchmarks and the potential for scaling to solve real-world problems, while underestimating its own belief in the importance of linguistic structure, inductive bias, and interdisciplinary science.


page 14

page 25

page 27

page 28

page 31


Meta Learning for Natural Language Processing: A Survey

Deep learning has been the mainstream technique in natural language proc...

Towards Countering Essentialism through Social Bias Reasoning

Essentialist beliefs (i.e., believing that members of the same group are...

Thorny Roses: Investigating the Dual Use Dilemma in Natural Language Processing

Dual use, the intentional, harmful reuse of technology and scientific ar...

Questioning the Survey Responses of Large Language Models

As large language models increase in capability, researchers have starte...

Towards transparency in NLP shared tasks

This article reports on a survey carried out across the Natural Language...

Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research

Many recent improvements in NLP stem from the development and use of lar...

Please sign up or login with your details

Forgot password? Click here to reset