Errors and secret data in the Italian research assessment exercise. A comment to a reply

07/21/2017
by   Alberto Baccini, et al.
0

Italy adopted a performance-based system for funding universities that is centered on the results of a national research assessment exercise, realized by a governmental agency (ANVUR). ANVUR evaluated papers by using 'a dual system of evaluation', that is by informed peer review or by bibliometrics. In view of validating that system, ANVUR performed an experiment for estimating the agreement between informed review and bibliometrics. Ancaiani et al. (2015) presents the main results of the experiment. Baccini and De Nicolao (2017) documented in a letter, among other critical issues, that the statistical analysis was not realized on a random sample of articles. A reply to the letter has been published by Research Evaluation (Benedetto et al. 2017). This note highlights that in the reply there are (1) errors in data, (2) problems with 'representativeness' of the sample, (3) unverifiable claims about weights used for calculating kappas, (4) undisclosed averaging procedures; (5) a statement about 'same protocol in all areas' contradicted by official reports. Last but not least: the data used by the authors continue to be undisclosed. A general warning concludes: many recently published papers use data originating from Italian research assessment exercise. These data are not accessible to the scientific community and consequently these papers are not reproducible. They can be hardly considered as containing sound evidence at least until authors or ANVUR disclose the data necessary for replication.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/13/2022

Analysis of publications by authors of Ukrainian institutes in Scopus-delisted titles

In Ukraine, Scopus data are used to evaluate academics. Existing shortco...
research
09/09/2022

Evaluating the (in)accessibility of data behind papers in astronomy

This paper presents results of a survey of authors of journal articles p...
research
11/05/2018

Evaluating research: from informed peer review to bibliometrics

National research assessment exercises are becoming regular events in ev...
research
03/26/2019

Like-for-like bibliometric substitutes for peer review: advantages and limits of indicators calculated from the ep index

The use of bibliometric indicators would simplify research assessments. ...
research
09/09/2022

On the Reliability of Funding Acknowledgements as Research Data: Evidence from Astronomy

Online bibliographic databases have enabled new research through which b...
research
12/23/2021

A Synthetic Prediction Market for Estimating Confidence in Published Work

Explainably estimating confidence in published scholarly work offers opp...

Please sign up or login with your details

Forgot password? Click here to reset