Convergent validity of several indicators measuring disruptiveness with milestone assignments to physics papers by experts

by   Lutz Bornmann, et al.

This study focuses on a recently introduced type of indicator measuring disruptiveness in science. Disruptive research diverges from current lines of research by opening up new lines. In the current study, we included the initially proposed indicator of this new type (Wu, Wang, Evans, 2019) and several variants with DI1: DI5, DI1n, DI5n, and DEP. Since indicators should measure what they propose to measure, we investigated the convergent validity of the indicators. We used a list of milestone papers, selected and published by editors of Physical Review Letters, and investigated whether this human (experts - based list is related to values of the several disruption indicators variants and - if so - which variants show the highest correlation with expert judgements. We used bivariate statistics, multiple regression models, and (coarsened) exact matching (CEM) to investigate the convergent validity of the indicators. The results show that the indicators correlate differently with the milestone paper assignments by the editors. It is not the initially proposed disruption index that performed best (DI1), but the variant DI5 which has been introduced by Bornmann, Devarakonda, Tekles, and Chacko (2019). In the CEM analysis of this study, the DEP variant - introduced by Bu, Waltman, and Huang (2019) - also showed favorable results.


page 15

page 20


Do disruption index indicators measure what they propose to measure? The comparison of several indicator variants with assessments by peers

Recently, Wu, Wang, and Evans (2019) and Bu, Waltman, and Huang (2019) p...

What do we know about the disruption indicator in scientometrics? An overview of the literature

The purpose of this paper is to provide a review of the literature on th...

How to measure research performance of single scientists? A proposal for an index based on scientific prizes: The Prize Winner Index (PWI)

In this study, we propose a new index for measuring excellence in scienc...

Like-for-like bibliometric substitutes for peer review: advantages and limits of indicators calculated from the ep index

The use of bibliometric indicators would simplify research assessments. ...

An empirical review of the different variants of the Probabilistic Affinity Index as applied to scientific collaboration

Responsible indicators are crucial for research assessment and monitorin...

Loss Rate Forecasting Framework Based on Macroeconomic Changes: Application to US Credit Card Industry

A major part of the balance sheets of the largest US banks consists of c...

Please sign up or login with your details

Forgot password? Click here to reset