CoDA21: Evaluating Language Understanding Capabilities of NLP Models With Context-Definition Alignment

03/11/2022
by   Lutfi Kerem Senel, et al.
7

Pretrained language models (PLMs) have achieved superhuman performance on many benchmarks, creating a need for harder tasks. We introduce CoDA21 (Context Definition Alignment), a challenging benchmark that measures natural language understanding (NLU) capabilities of PLMs: Given a definition and a context each for k words, but not the words themselves, the task is to align the k definitions with the k contexts. CoDA21 requires a deep understanding of contexts and definitions, including complex inference and world knowledge. We find that there is a large gap between human and PLM performance, suggesting that CoDA21 measures an aspect of NLU that is not sufficiently covered in existing benchmarks.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset