Predicting article quality scores with machine learning: The UK Research Excellence Framework

12/11/2022
by   Mike Thelwall, et al.
0

National research evaluation initiatives and incentive schemes have previously chosen between simplistic quantitative indicators and time-consuming peer review, sometimes supported by bibliometrics. Here we assess whether artificial intelligence (AI) could provide a third alternative, estimating article quality using more multiple bibliometric and metadata inputs. We investigated this using provisional three-level REF2021 peer review scores for 84,966 articles submitted to the UK Research Excellence Framework 2021, matching a Scopus record 2014-18 and with a substantial abstract. We found that accuracy is highest in the medical and physical sciences Units of Assessment (UoAs) and economics, reaching 42 case. This is based on 1000 bibliometric inputs and half of the articles used for training in each UoA. Prediction accuracies above the baseline for the social science, mathematics, engineering, arts, and humanities UoAs were much lower or close to zero. The Random Forest Classifier (standard or ordinal) and Extreme Gradient Boosting Classifier algorithms performed best from the 32 tested. Accuracy was lower if UoAs were merged or replaced by Scopus broad categories. We increased accuracy with an active learning strategy and by selecting articles with higher prediction probabilities, as estimated by the algorithms, but this substantially reduced the number of scores predicted.

READ FULL TEXT

page 11

page 13

research
12/11/2022

Do bibliometrics introduce gender, institutional or interdisciplinary biases into research evaluations?

Systematic evaluations of publicly funded research typically employ a co...
research
12/11/2022

Are internationally co-authored journal articles better quality? The UK case 2014-2020

International collaboration is sometimes encouraged in the belief that i...
research
12/11/2022

Artificial intelligence technologies to support research assessment: A review

This literature review identifies indicators that associate with higher ...
research
10/31/2018

National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case

There has been ample demonstration that bibliometrics is superior to pee...
research
12/11/2022

Why are co-authored academic articles more cited: Higher quality or larger audience?

Co-authored articles tend to be more cited in many academic fields, but ...
research
12/11/2022

Is Research Funding Always Beneficial? A Cross-Disciplinary Analysis of UK Research 2014-20

The search for and management of external funding now occupies much valu...
research
03/11/2019

The rhetorical structure of science? A multidisciplinary analysis of article headings

An effective structure helps an article to convey its core message. The ...

Please sign up or login with your details

Forgot password? Click here to reset