Do open citations inform the qualitative peer-review evaluation in research assessments? An analysis of the Italian National Scientific Qualification

by   Federica Bologna, et al.

In the past, several works have investigated ways for combining quantitative and qualitative methods in research assessment exercises. Indeed, the Italian National Scientific Qualification (NSQ), i.e. the national assessment exercise which aims at deciding whether a scholar can apply to professorial academic positions as Associate Professor and Full Professor, adopts a quantitative and qualitative evaluation process: it makes use of bibliometrics followed by a peer-review process of candidates' CVs. The NSQ divides academic disciplines into two categories, i.e. citation-based disciplines (CDs) and non-citation-based disciplines (NDs), a division that affects the metrics used for assessing the candidates of that discipline in the first part of the process, which is based on bibliometrics. In this work, we aim at exploring whether citation-based metrics, calculated only considering open bibliographic and citation data, can support the human peer-review of NDs and yield insights on how it is conducted. To understand if and what citation-based (and, possibly, other) metrics provide relevant information, we created a series of machine learning models to replicate the decisions of the NSQ committees. As one of the main outcomes of our study, we noticed that the strength of the citational relationship between the candidate and the commission in charge of assessing his/her CV seems to play a role in the peer-review phase of the NSQ of NDs.


page 11

page 15

page 23

page 24

page 26


Academics evaluating academics: a methodology to inform the review process on top of open citations

In the past, several works have investigated ways for combining quantita...

Overton – A bibliometric database of policy document citations

This paper presents an analysis of the Overton policy document database,...

Peer review and citation data in predicting university rankings, a large-scale analysis

Most Performance-based Research Funding Systems (PRFS) draw on peer revi...

Cite-seeing and Reviewing: A Study on Citation Bias in Peer Review

Citations play an important role in researchers' careers as a key factor...

Who Owns the Data? A Systematic Review at the Boundary of Information Systems and Marketing

This paper gives a systematic research review at the boundary of the inf...

Peer-review under review - A statistical study on proposal ranking at ESO. Part I: the pre-meeting phase

Peer review is the most common mechanism in place for assessing requests...

Content-Based Citation Recommendation

We present a content-based method for recommending citations in an acade...

Please sign up or login with your details

Forgot password? Click here to reset