Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data

by   Lutz Bornmann, et al.

Lee, Walsh, and Wang (2015) - based on Uzzi, Mukherjee, Stringer, and Jones (2013) - and Wang, Veugelers, and Stephan (2017) proposed scores based on cited references (cited journals) data which can be used to measure the novelty of papers (named as novelty scores U and W in this study). Although previous research has used novelty scores in various empirical analyses, no study has been published up to now - to the best of our knowledge - which quantitatively tested the convergent validity of novelty scores: do these scores measure what they propose to measure? Using novelty assessments by faculty members (FMs) at F1000Prime for comparison, we tested the convergent validity of the two novelty scores (U and W). FMs' assessments not only refer to the quality of biomedical papers, but also to their characteristics (by assigning certain tags to the papers): for example, are the presented findings or formulated hypotheses novel (tags "new findings" and "hypothesis")? We used these and other tags to investigate the convergent validity of both novelty scores. Our study reveals different results for the novelty scores: the results for novelty score U are mostly in agreement with previously formulated expectations. We found, for instance, that for a standard deviation (one unit) increase in novelty score U, the expected number of assignments of the "new finding" tag increase by 7.47 The results for novelty score W, however, do not reflect convergent validity with the FMs' assessments: only the results for some tags are in agreement with the expectations. Thus, we propose - based on our results - the use of novelty score U for measuring novelty quantitatively, but question the use of novelty score W.


page 1

page 2

page 3

page 4


Do disruption index indicators measure what they propose to measure? The comparison of several indicator variants with assessments by peers

Recently, Wu, Wang, and Evans (2019) and Bu, Waltman, and Huang (2019) p...

Creativity in Science and the Link to Cited References: Is the Creative Potential of Papers Reflected in their Cited References?

Several authors have proposed that a large number of unusual combination...

Convergent validity of several indicators measuring disruptiveness with milestone assignments to physics papers by experts

This study focuses on a recently introduced type of indicator measuring ...

Limiting Tags Fosters Efficiency

Tagging facilitates information retrieval in social media and other onli...

Novelty Search in Competitive Coevolution

One of the main motivations for the use of competitive coevolution syste...

Novelty and Foreseeing Research Trends; The Case of Astrophysics and Astronomy

Metrics based on reference lists of research articles or on keywords hav...

FinTech E-Commerce Payment Application User Experience Analysis during COVID-19 Pandemic

The application of information technology in the era of big data and clo...

Please sign up or login with your details

Forgot password? Click here to reset