BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?

by   Asahi Ushio, et al.

Analogies play a central role in human commonsense reasoning. The ability to recognize analogies such as eye is to seeing what ear is to hearing, sometimes referred to as analogical proportions, shape how we structure knowledge and understand language. Surprisingly, however, the task of identifying such analogies has not yet received much attention in the language model era. In this paper, we analyze the capabilities of transformer-based language models on this unsupervised task, using benchmarks obtained from educational settings, as well as more commonly used datasets. We find that off-the-shelf language models can identify analogies to a certain extent, but struggle with abstract and complex relations, and results are highly sensitive to model architecture and hyperparameters. Overall the best results were obtained with GPT-2 and RoBERTa, while configurations using BERT were not able to outperform word embedding models. Our results raise important questions for future work about how, and to what extent, pre-trained language models capture knowledge about abstract semantic relations[Source code and data to reproduce our experimental results are available in the following repository: <>].


Probing for Bridging Inference in Transformer Language Models

We probe pre-trained transformer language models for bridging inference....

Is the Computation of Abstract Sameness Relations Human-Like in Neural Language Models?

In recent years, deep neural language models have made strong progress i...

Do Language Models Know the Way to Rome?

The global geometry of language models is important for a range of appli...

AstBERT: Enabling Language Model for Code Understanding with Abstract Syntax Tree

Using a pre-trained language model (i.e. BERT) to apprehend source codes...

Multilinguals at SemEval-2022 Task 11: Transformer Based Architecture for Complex NER

We investigate the task of complex NER for the English language. The tas...

The Role of Complex NLP in Transformers for Text Ranking?

Even though term-based methods such as BM25 provide strong baselines in ...

How is BERT surprised? Layerwise detection of linguistic anomalies

Transformer language models have shown remarkable ability in detecting w...

Code Repositories


The official implementation of BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies? which has been accepted by ACL 2021 main conference.

view repo