A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions

11/30/2020
by   Ivan Stelmakh, et al.
0

Peer review is the backbone of academia and humans constitute a cornerstone of this process, being responsible for reviewing papers and making the final acceptance/rejection decisions. Given that human decision making is known to be susceptible to various cognitive biases, it is important to understand which (if any) biases are present in the peer-review process and design the pipeline such that the impact of these biases is minimized. In this work, we focus on the dynamics of between-reviewers discussions and investigate the presence of herding behaviour therein. In that, we aim to understand whether reviewers and more senior decision makers get disproportionately influenced by the first argument presented in the discussion when (in case of reviewers) they form an independent opinion about the paper before discussing it with others. Specifically, in conjunction with the review process of ICML 2020 – a large, top tier machine learning conference – we design and execute a randomized controlled trial with the goal of testing for the conditional causal effect of the discussion initiator's opinion on the outcome of a paper.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/29/2018

Unifying Decision-Making: a Review on Evolutionary Theories on Rationality and Cognitive Biases

In this paper, we make a review on the concepts of rationality across se...
research
12/31/2019

On Testing for Biases in Peer Review

We consider the issue of biases in scholarly research, specifically, in ...
research
10/09/2020

Large-scale randomized experiment reveals machine learning helps people learn and remember more effectively

Machine learning has typically focused on developing models and algorith...
research
07/11/2023

Testing for Reviewer Anchoring in Peer Review: A Randomized Controlled Trial

Peer review frequently follows a process where reviewers first provide i...
research
11/30/2020

Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review

Modern machine learning and computer science conferences are experiencin...
research
02/07/2023

Simulating the impact of cognitive biases on the mobility transition

Climate change is becoming more visible, and human adaptation is require...
research
06/24/2023

Estimating the Causal Effect of Early ArXiving on Paper Acceptance

What is the effect of releasing a preprint of a paper before it is submi...

Please sign up or login with your details

Forgot password? Click here to reset