Evaluating the Efficacy of ChatGPT-4 in Providing Scientific References across Diverse Disciplines

by   Zhi Cao, et al.

This work conducts a comprehensive exploration into the proficiency of OpenAI's ChatGPT-4 in sourcing scientific references within an array of research disciplines. Our in-depth analysis encompasses a wide scope of fields including Computer Science (CS), Mechanical Engineering (ME), Electrical Engineering (EE), Biomedical Engineering (BME), and Medicine, as well as their more specialized sub-domains. Our empirical findings indicate a significant variance in ChatGPT-4's performance across these disciplines. Notably, the validity rate of suggested articles in CS, BME, and Medicine surpasses 65 whereas in the realms of ME and EE, the model fails to verify any article as valid. Further, in the context of retrieving articles pertinent to niche research topics, ChatGPT-4 tends to yield references that align with the broader thematic areas as opposed to the narrowly defined topics of interest. This observed disparity underscores the pronounced variability in accuracy across diverse research fields, indicating the potential requirement for model refinement to enhance its functionality in academic research. Our investigation offers valuable insights into the current capacities and limitations of AI-powered tools in scholarly research, thereby emphasizing the indispensable role of human oversight and rigorous validation in leveraging such models for academic pursuits.


Analyzing the State of Computer Science Research with the DBLP Discovery Dataset

The number of scientific publications continues to rise exponentially, e...

Digital Health Discussion Through Articles Published Until the Year 2021: A Digital Topic Modeling Approach

The digital health industry has grown in popularity since the 2010s, but...

XFlow: Benchmarking Flow Behaviors over Graphs

The occurrence of diffusion on a graph is a prevalent and significant ph...

Citing and referencing habits in Medicine and Social Sciences journals in 2019

This article explores citing and referencing systems in Social Sciences ...

Mapping Computer Science Research: Trends, Influences, and Predictions

This paper explores the current trending research areas in the field of ...

Tracking biomedical articles along the translational continuum: a measure based on biomedical knowledge representation

Keeping track of translational research is essential to evaluating the p...

Please sign up or login with your details

Forgot password? Click here to reset