Mitigating Skewed Bidding for Conference Paper Assignment

03/01/2023
by   Inbal Rozencweig, et al.
0

The explosion of conference paper submissions in AI and related fields, has underscored the need to improve many aspects of the peer review process, especially the matching of papers and reviewers. Recent work argues that the key to improve this matching is to modify aspects of the bidding phase itself, to ensure that the set of bids over papers is balanced, and in particular to avoid orphan papers, i.e., those papers that receive no bids. In an attempt to understand and mitigate this problem, we have developed a flexible bidding platform to test adaptations to the bidding process. Using this platform, we performed a field experiment during the bidding phase of a medium-size international workshop that compared two bidding methods. We further examined via controlled experiments on Amazon Mechanical Turk various factors that affect bidding, in particular the order in which papers are presented <cit.>; and information on paper demand <cit.>. Our results suggest that several simple adaptations, that can be added to any existing platform, may significantly reduce the skew in bids, thereby improving the allocation for both reviewers and conference organizers.

READ FULL TEXT
research
12/11/2018

Proceedings of the 17th Dutch-Belgian Information Retrieval Workshop

This volume contains the papers presented at DIR 2018: 17th Dutch-Belgia...
research
05/13/2023

Reviewer assignment problem: A scoping review

Peer review is an integral component of scientific research. The quality...
research
09/20/2021

Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment

In this paper we revisit the 2014 NeurIPS experiment that examined incon...
research
05/27/2023

Counterfactual Evaluation of Peer-Review Assignment Policies

Peer review assignment algorithms aim to match research papers to suitab...
research
08/02/2023

Summary of 2nd International Workshop on Requirements Engineering and Testing (RET)

The RET (Requirements Engineering and Testing) workshop series provides ...
research
05/26/2016

cvpaper.challenge in 2015 - A review of CVPR2015 and DeepSurvey

The "cvpaper.challenge" is a group composed of members from AIST, Tokyo ...
research
08/13/2021

Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design

Many scientific conferences employ a two-phase paper review process, whe...

Please sign up or login with your details

Forgot password? Click here to reset