Like-for-like bibliometric substitutes for peer review: advantages and limits of indicators calculated from the ep index

by   Alonso Rodriguez-Navarro, et al.

The use of bibliometric indicators would simplify research assessments. The 2014 Research Excellence Framework (REF) is a peer review assessment of UK universities, whose results can be taken as benchmarks for bibliometric indicators. In this study we use the REF results to investigate whether the ep index and a top percentile of most cited papers could substitute for peer review. The probability that a random university's paper reaches a certain top percentile in the global distribution of papers is a power of the ep index, which can be calculated from the citation-based distribution of university's papers in global top percentiles. Making use of the ep index in each university and research area, we calculated the ratios between the percentage of 4-star-rated outputs in REF and the percentages of papers in global top percentiles. Then, we fixed the assessment percentile so that the mean ratio between these two indicators across universities is 1.0. This method was applied to four units of assessment in REF: Chemistry, Economics Econometrics joined to Business Management Studies, and Physics. Some relevant deviations from the 1.0 ratio could be explained by the evaluation procedure in REF or by the characteristics of the research field; other deviations need specific studies by experts in the research area. The present results indicate that in many research areas the substitution of a top percentile indicator for peer review is possible. However, this substitution cannot be made straightforwardly; more research is needed to establish the conditions of the bibliometric assessment.


Peer review and citation data in predicting university rankings, a large-scale analysis

Most Performance-based Research Funding Systems (PRFS) draw on peer revi...

National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case

There has been ample demonstration that bibliometrics is superior to pee...

Do disruption index indicators measure what they propose to measure? The comparison of several indicator variants with assessments by peers

Recently, Wu, Wang, and Evans (2019) and Bu, Waltman, and Huang (2019) p...

An empirical review of the different variants of the Probabilistic Affinity Index as applied to scientific collaboration

Responsible indicators are crucial for research assessment and monitorin...

Combating harmful Internet use with peer assessment and differential evolution

Harmful Internet use (HIU) is a term coined for the unintended use of th...

Convergent validity of several indicators measuring disruptiveness with milestone assignments to physics papers by experts

This study focuses on a recently introduced type of indicator measuring ...

Errors and secret data in the Italian research assessment exercise. A comment to a reply

Italy adopted a performance-based system for funding universities that i...

Please sign up or login with your details

Forgot password? Click here to reset