research
∙
07/11/2023
Testing for Reviewer Anchoring in Peer Review: A Randomized Controlled Trial
Peer review frequently follows a process where reviewers first provide i...
share
research
∙
06/01/2023
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Given the rapid ascent of large language models (LLMs), we study the que...
research
∙
05/27/2023
Counterfactual Evaluation of Peer-Review Assignment Policies
Peer review assignment algorithms aim to match research papers to suitab...
research
∙
03/23/2023
A Gold Standard Dataset for the Reviewer Assignment Problem
Many peer-review venues are either using or looking to use algorithms to...
research
∙
02/16/2023
Assisting Human Decisions in Document Matching
Many practical applications, ranging from paper-reviewer assignment in p...
research
∙
12/31/2022
The Role of Author Identities in Peer Review
There is widespread debate on whether to anonymize author identities in ...
research
∙
11/23/2022
Batching of Tasks by Users of Pseudonymous Forums: Anonymity Compromise and Protection
There are a number of forums where people participate under pseudonyms. ...
research
∙
11/22/2022
How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?
How do author perceptions match up to the outcomes of the peer-review pr...
research
∙
09/18/2022
Allocation Schemes in Analytic Evaluation: Applicant-Centric Holistic or Attribute-Centric Segmented?
Many applications such as hiring and university admissions involve evalu...
research
∙
07/22/2022
Tradeoffs in Preventing Manipulation in Paper Bidding for Reviewer Assignment
Many conferences rely on paper bidding as a key component of their revie...
research
∙
06/24/2022
A Dataset on Malicious Paper Bidding in Peer Review
In conference peer review, reviewers are often asked to provide "bids" o...
research
∙
04/05/2022
Integrating Rankings into Quantized Scores in Peer Review
In peer review, reviewers are usually asked to provide scores for the pa...
research
∙
03/31/2022
To ArXiv or not to ArXiv: A Study Quantifying Pros and Cons of Posting Preprints Online
Double-blind conferences have engaged in debates over whether to allow a...
research
∙
03/31/2022
Cite-seeing and Reviewing: A Study on Citation Bias in Peer Review
Citations play an important role in researchers' careers as a key factor...
research
∙
01/27/2022
Calibration with Privacy in Peer Review
Reviewers in peer review are often miscalibrated: they may be strict, le...
research
∙
01/25/2022
Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators' Expertise
Strategic behavior is a fundamental problem in a variety of real-world a...
research
∙
08/13/2021
Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design
Many scientific conferences employ a two-phase paper review process, whe...
research
∙
12/01/2020
Debiasing Evaluations That are Biased by Evaluations
It is common to evaluate a set of items by soliciting people to rate the...
research
∙
11/30/2020
A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions
Peer review is the backbone of academia and humans constitute a cornerst...
research
∙
11/30/2020
A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences
Conference peer review constitutes a human-computation process whose imp...
research
∙
11/30/2020
Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review
Modern machine learning and computer science conferences are experiencin...
research
∙
10/29/2020
Uncovering Latent Biases in Text: Method and Application to Peer Review
Quantifying systematic disparities in numerical quantities such as emplo...
research
∙
10/08/2020
Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment
We consider the issue of strategic behaviour in various peer-assessment ...
research
∙
06/29/2020
Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments
We consider three important challenges in conference peer review: (i) re...
research
∙
06/29/2020
On the Privacy-Utility Tradeoff in Peer-Review Data Analysis
A major impediment to research on improving peer review is the unavailab...
research
∙
06/27/2020
A SUPER* Algorithm to Optimize Paper Bidding in Peer Review
A number of applications involve sequential arrival of users, and requir...
research
∙
06/21/2020
Two-Sample Testing on Ranked Preference Data and the Role of Modeling Assumptions
A number of applications require two-sample testing on ranked preference...
research
∙
12/31/2019
On Testing for Biases in Peer Review
We consider the issue of biases in scholarly research, specifically, in ...
research
∙
06/10/2019
Stretching the Effectiveness of MLE from Accuracy to Bias for Pairwise Comparisons
A number of applications (e.g., AI bot tournaments, sports, peer grading...
research
∙
09/13/2018
An Incentive Mechanism for Crowd Sensing with Colluding Agents
Vehicular mobile crowd sensing is a fast-emerging paradigm to collect da...
research
∙
08/27/2018
Choosing How to Choose Papers
It is common to see a handful of reviewers reject a highly novel paper, ...
research
∙
06/16/2018
On Strategyproof Conference Peer Review
We consider peer review in a conference setting where there is typically...
research
∙
06/16/2018
PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review
We consider the problem of automated assignment of papers to reviewers i...
research
∙
06/13/2018
Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings
Cardinal scores (numeric ratings) collected from people are well known t...
research
∙
09/01/2017
Low Permutation-rank Matrices: Structural Properties and Noisy Completion
We consider the problem of noisy matrix completion, in which the goal is...
research
∙
08/31/2017
Design and Analysis of the NIPS 2016 Review Process
Neural Information Processing Systems (NIPS) is a top-tier annual confer...
research
∙
06/30/2016
A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness
The aggregation and denoising of crowd labeled data is a task that has g...
research
∙
06/28/2016
Active Ranking from Pairwise Comparisons and when Parametric Assumptions Don't Help
We consider sequential or active ranking of a set of n items based on no...
research
∙
03/22/2016
Feeling the Bern: Adaptive Estimators for Bernoulli Probabilities of Pairwise Comparisons
We study methods for aggregating pairwise comparison data in order to es...
research
∙
02/24/2016
Parametric Prediction from Parametric Agents
We consider a problem of prediction based on opinions elicited from hete...
research
∙
12/30/2015
Simple, Robust and Optimal Ranking from Pairwise Comparisons
We consider data in the form of pairwise comparisons of n items, with th...
research
∙
10/19/2015
Stochastically Transitive Models for Pairwise Comparisons: Statistical and Computational Issues
There are various parametric models for analyzing pairwise comparison da...
research
∙
05/06/2015
Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence
Data in the form of pairwise comparisons arises in many domains, includi...
research
∙
03/25/2015
Regularized Minimax Conditional Entropy for Crowdsourcing
There is a rapidly increasing interest in crowdsourcing for data labelin...
research
∙
02/19/2015
Approval Voting and Incentives in Crowdsourcing
The growing need for labeled training data has made crowdsourcing an imp...
research
∙
11/21/2014
On the Impossibility of Convex Inference in Human Computation
Human computation or crowdsourcing involves joint inference of the groun...
research
∙
06/25/2014