What do we know about the disruption indicator in scientometrics? An overview of the literature

by   Christian Leibel, et al.

The purpose of this paper is to provide a review of the literature on the original disruption indicator (DI1) and its variants in scientometrics. The DI1 has received much media attention and prompted a public debate about science policy implications, since a study published in Nature found that papers in all disciplines and patents are becoming less disruptive over time. This review explains in the first part the DI1 and its variants in detail by examining their technical and theoretical properties. The remaining parts of the review are devoted to studies that examine the validity and the limitations of the indicators. Particular focus is placed on (1) the time-sensitivity of disruption scores, (2) the convergent validity of disruption scores with expert judgments, and (3) the comparative performance of the DI1 and its variants. The review shows that, while the literature on convergent validity is not entirely conclusive, it is clear that some modified indicator variants, in particular DI5, show higher degrees of convergent validity than DI1. Limitations of the DI1 and its variants are summarized, and best practice guidelines are provided. The review encourages users of the indicator to inform about the variety of DI1 variants and to apply the most appropriate variant. More research on the convergent validity of the DI1 and its variants as well as on the time-sensitivity of disruption scores is needed before the indicators can be used in the research evaluation practice.


page 12

page 20


Convergent validity of several indicators measuring disruptiveness with milestone assignments to physics papers by experts

This study focuses on a recently introduced type of indicator measuring ...

An empirical review of the different variants of the Probabilistic Affinity Index as applied to scientific collaboration

Responsible indicators are crucial for research assessment and monitorin...

Do disruption index indicators measure what they propose to measure? The comparison of several indicator variants with assessments by peers

Recently, Wu, Wang, and Evans (2019) and Bu, Waltman, and Huang (2019) p...

What is Legitimate Decision Support?

Decision support is the science and associated practice that consist in ...

Reviewing, indicating, and counting books for modern research evaluation systems

In this chapter, we focus on the specialists who have helped to improve ...

Please sign up or login with your details

Forgot password? Click here to reset