Exploration of reproducibility issues in scientometric research Part 1: Direct reproducibility

04/13/2018
by   Ludo Waltman, et al.
0

This is the first part of a small-scale explorative study in an effort to start assessing reproducibility issues specific to scientometrics research. This effort is motivated by the desire to generate empirical data to inform debates about reproducibility in scientometrics. Rather than attempt to reproduce studies, we explore how we might assess "in principle" reproducibility based on a critical review of the content of published papers. The first part of the study focuses on direct reproducibility - that is the ability to reproduce the specific evidence produced by an original study using the same data, methods, and procedures. The second part (Velden et al. 2018) is dedicated to conceptual reproducibility - that is the robustness of knowledge claims towards verification by an alternative approach using different data, methods and procedures. The study is exploratory: it investigates only a very limited number of publications and serves us to develop instruments for identifying potential reproducibility issues of published studies: These are a categorization of study types and a taxonomy of threats to reproducibility. We work with a select sample of five publications in scientometrics covering a variation of study types of theoretical, methodological, and empirical nature. Based on observations made during our exploratory review, we conclude this paper with open questions on how to approach and assess the status of direct reproducibility in scientometrics, intended for discussion at the special track on "Reproducibility in Scientometrics" at STI2018 in Leiden.

READ FULL TEXT
research
04/13/2018

Exploration of Reproducibility Issues in Scientometric Research Part 2: Conceptual Reproducibility

This is the second part of a small-scale explorative study in an effort ...
research
09/30/2022

The state of play of reproducibility in Statistics: an empirical analysis

Reproducibility, the ability to reproduce the results of published paper...
research
04/10/2019

Applicability study of the PRIMAD model to LIGO gravitational wave search workflows

The PRIMAD model with its six components (i.e., Platform, Research Objec...
research
02/09/2018

Terminologies for Reproducible Research

Reproducible research---by its many names---has come to be regarded as a...
research
04/20/2023

A Study on Reproducibility and Replicability of Table Structure Recognition Methods

Concerns about reproducibility in artificial intelligence (AI) have emer...
research
09/09/2022

Computational reproducibility of Jupyter notebooks from biomedical publications

Jupyter notebooks allow to bundle executable code with its documentation...
research
12/19/2021

Reproducibility Challenges and Their Impacts on Technical Q A Websites: The Practitioners' Perspectives

Software developers often look for solutions to their code-level problem...

Please sign up or login with your details

Forgot password? Click here to reset