AI Chat AI Image Generator AI Video Text to Speech

The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through

03/29/2022

∙

by Shruti Singh, et al.

∙

∙

Language models are increasingly becoming popular in AI-powered scientific IR systems. This paper evaluates popular scientific language models in handling (i) short-query texts and (ii) textual neighbors. Our experiments showcase the inability to retrieve relevant documents for a short-query text even under the most relaxed conditions. Additionally, we leverage textual neighbors, generated by small perturbations to the original text, to demonstrate that not all perturbations lead to close neighbors in the embedding space. Further, an exhaustive categorization yields several classes of orthographically and semantically related, partially related, and completely unrelated neighbors. Retrieval performance turns out to be more influenced by the surface form rather than the semantics of the text.

Shruti Singh
6 publications
Mayank Singh
47 publications

page 15

page 19

research

∙ 05/29/2023

Test-Time Training on Nearest Neighbors for Large Language Models

Many recent efforts aim to augment language models with relevant informa...

0 Moritz Hardt, et al. ∙

research

∙ 02/18/2022

Modelling the semantics of text in complex document layouts using graph transformer networks

Representing structured text from complex documents typically calls for ...

0 Thomas Roland Barillot, et al. ∙

research

∙ 05/25/2023

Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models

Augmenting language models with a retrieval mechanism has been shown to ...

0 Ehsan Doostmohammadi, et al. ∙

research

∙ 05/03/2022

SemAttack: Natural Textual Attacks via Different Semantic Spaces

Recent studies show that pre-trained language models (LMs) are vulnerabl...

0 Boxin Wang, et al. ∙

research

∙ 04/11/2023

Emergent autonomous scientific research capabilities of large language models

Transformer-based large language models are rapidly advancing in the fie...

0 Daniil A. Boiko, et al. ∙

research

∙ 06/08/2023

Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding

Large Language Models (LLMs) have been applied in the speech domain, oft...

0 Mingqiu Wang, et al. ∙

research

∙ 10/28/2022

You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LM

Retrieval-enhanced language models (LMs), which condition their predicti...

0 Andrew Drozdov, et al. ∙