b'Nihar B. Shah'

research

∙ 07/11/2023

Testing for Reviewer Anchoring in Peer Review: A Randomized Controlled Trial

Peer review frequently follows a process where reviewers first provide i...

0 Ryan Liu, et al. ∙

research

∙ 06/01/2023

ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing

Given the rapid ascent of large language models (LLMs), we study the que...

0 Ryan Liu, et al. ∙

research

∙ 05/27/2023

Counterfactual Evaluation of Peer-Review Assignment Policies

Peer review assignment algorithms aim to match research papers to suitab...

0 Martin Saveski, et al. ∙

research

∙ 03/23/2023

A Gold Standard Dataset for the Reviewer Assignment Problem

Many peer-review venues are either using or looking to use algorithms to...

0 Ivan Stelmakh, et al. ∙

research

∙ 02/16/2023

Assisting Human Decisions in Document Matching

Many practical applications, ranging from paper-reviewer assignment in p...

0 Joon Sik Kim, et al. ∙

research

∙ 12/31/2022

The Role of Author Identities in Peer Review

There is widespread debate on whether to anonymize author identities in ...

0 Nihar B. Shah, et al. ∙

research

∙ 11/23/2022

Batching of Tasks by Users of Pseudonymous Forums: Anonymity Compromise and Protection

There are a number of forums where people participate under pseudonyms. ...

0 Alexander Goldberg, et al. ∙

research

∙ 11/22/2022

How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?

How do author perceptions match up to the outcomes of the peer-review pr...

0 Charvi Rastogi, et al. ∙

research

∙ 09/18/2022

Allocation Schemes in Analytic Evaluation: Applicant-Centric Holistic or Attribute-Centric Segmented?

Many applications such as hiring and university admissions involve evalu...

0 Jingyan Wang, et al. ∙

research

∙ 07/22/2022

Tradeoffs in Preventing Manipulation in Paper Bidding for Reviewer Assignment

Many conferences rely on paper bidding as a key component of their revie...

0 Steven Jecmen, et al. ∙

research

∙ 06/24/2022

A Dataset on Malicious Paper Bidding in Peer Review

In conference peer review, reviewers are often asked to provide "bids" o...

0 Steven Jecmen, et al. ∙

research

∙ 04/05/2022

Integrating Rankings into Quantized Scores in Peer Review

In peer review, reviewers are usually asked to provide scores for the pa...

0 Yusha Liu, et al. ∙

research

∙ 03/31/2022

To ArXiv or not to ArXiv: A Study Quantifying Pros and Cons of Posting Preprints Online

Double-blind conferences have engaged in debates over whether to allow a...

0 Charvi Rastogi, et al. ∙

research

∙ 03/31/2022

Cite-seeing and Reviewing: A Study on Citation Bias in Peer Review

Citations play an important role in researchers' careers as a key factor...

0 Ivan Stelmakh, et al. ∙

research

∙ 01/27/2022

Calibration with Privacy in Peer Review

Reviewers in peer review are often miscalibrated: they may be strict, le...

0 Wenxin Ding, et al. ∙

research

∙ 01/25/2022

Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators' Expertise

Strategic behavior is a fundamental problem in a variety of real-world a...

0 Komal Dhull, et al. ∙

research

∙ 08/13/2021

Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design

Many scientific conferences employ a two-phase paper review process, whe...

0 Steven Jecmen, et al. ∙

research

∙ 12/01/2020

Debiasing Evaluations That are Biased by Evaluations

It is common to evaluate a set of items by soliciting people to rate the...

0 Jingyan Wang, et al. ∙

research

∙ 11/30/2020

A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions

Peer review is the backbone of academia and humans constitute a cornerst...

0 Ivan Stelmakh, et al. ∙

research

∙ 11/30/2020

A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences

Conference peer review constitutes a human-computation process whose imp...

0 Ivan Stelmakh, et al. ∙

research

∙ 11/30/2020

Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review

Modern machine learning and computer science conferences are experiencin...

0 Ivan Stelmakh, et al. ∙

research

∙ 10/29/2020

Uncovering Latent Biases in Text: Method and Application to Peer Review

Quantifying systematic disparities in numerical quantities such as emplo...

0 Emaad Manzoor, et al. ∙

research

∙ 10/08/2020

Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment

We consider the issue of strategic behaviour in various peer-assessment ...

0 Ivan Stelmakh, et al. ∙

research

∙ 06/29/2020

Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments

We consider three important challenges in conference peer review: (i) re...

41 Steven Jecmen, et al. ∙

research

∙ 06/29/2020

On the Privacy-Utility Tradeoff in Peer-Review Data Analysis

A major impediment to research on improving peer review is the unavailab...

0 Wenxin Ding, et al. ∙

research

∙ 06/27/2020

A SUPER* Algorithm to Optimize Paper Bidding in Peer Review

A number of applications involve sequential arrival of users, and requir...

0 Tanner Fiez, et al. ∙

research

∙ 06/21/2020

Two-Sample Testing on Ranked Preference Data and the Role of Modeling Assumptions

A number of applications require two-sample testing on ranked preference...

0 Charvi Rastogi, et al. ∙

research

∙ 12/31/2019

On Testing for Biases in Peer Review

We consider the issue of biases in scholarly research, specifically, in ...

0 Ivan Stelmakh, et al. ∙

research

∙ 06/10/2019

Stretching the Effectiveness of MLE from Accuracy to Bias for Pairwise Comparisons

A number of applications (e.g., AI bot tournaments, sports, peer grading...

0 Jingyan Wang, et al. ∙

research

∙ 09/13/2018

An Incentive Mechanism for Crowd Sensing with Colluding Agents

Vehicular mobile crowd sensing is a fast-emerging paradigm to collect da...

0 Susu Xu, et al. ∙

research

∙ 08/27/2018

Choosing How to Choose Papers

It is common to see a handful of reviewers reject a highly novel paper, ...

0 Ritesh Noothigattu, et al. ∙

research

∙ 06/16/2018

On Strategyproof Conference Peer Review

We consider peer review in a conference setting where there is typically...

0 Yichong Xu, et al. ∙

research

∙ 06/16/2018

PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review

We consider the problem of automated assignment of papers to reviewers i...

0 Ivan Stelmakh, et al. ∙

research

∙ 06/13/2018

Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings

Cardinal scores (numeric ratings) collected from people are well known t...

0 Jingyan Wang, et al. ∙

research

∙ 09/01/2017

Low Permutation-rank Matrices: Structural Properties and Noisy Completion

We consider the problem of noisy matrix completion, in which the goal is...

0 Nihar B. Shah, et al. ∙

research

∙ 08/31/2017

Design and Analysis of the NIPS 2016 Review Process

Neural Information Processing Systems (NIPS) is a top-tier annual confer...

0 Nihar B. Shah, et al. ∙

research

∙ 06/30/2016

A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness

The aggregation and denoising of crowd labeled data is a task that has g...

0 Nihar B. Shah, et al. ∙

research

∙ 06/28/2016

Active Ranking from Pairwise Comparisons and when Parametric Assumptions Don't Help

We consider sequential or active ranking of a set of n items based on no...

0 Reinhard Heckel, et al. ∙

research

∙ 03/22/2016

Feeling the Bern: Adaptive Estimators for Bernoulli Probabilities of Pairwise Comparisons

We study methods for aggregating pairwise comparison data in order to es...

0 Nihar B. Shah, et al. ∙

research

∙ 02/24/2016

Parametric Prediction from Parametric Agents

We consider a problem of prediction based on opinions elicited from hete...

0 Yuan Luo, et al. ∙

research

∙ 12/30/2015

Simple, Robust and Optimal Ranking from Pairwise Comparisons

We consider data in the form of pairwise comparisons of n items, with th...

0 Nihar B. Shah, et al. ∙

research

∙ 10/19/2015

Stochastically Transitive Models for Pairwise Comparisons: Statistical and Computational Issues

There are various parametric models for analyzing pairwise comparison da...

0 Nihar B. Shah, et al. ∙

research

∙ 05/06/2015

Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence

Data in the form of pairwise comparisons arises in many domains, includi...

0 Nihar B. Shah, et al. ∙

research

∙ 03/25/2015

Regularized Minimax Conditional Entropy for Crowdsourcing

There is a rapidly increasing interest in crowdsourcing for data labelin...

0 Dengyong Zhou, et al. ∙

research

∙ 02/19/2015

Approval Voting and Incentives in Crowdsourcing

The growing need for labeled training data has made crowdsourcing an imp...

0 Nihar B. Shah, et al. ∙

research

∙ 11/21/2014

On the Impossibility of Convex Inference in Human Computation

Human computation or crowdsourcing involves joint inference of the groun...

0 Nihar B. Shah, et al. ∙

research

∙ 06/25/2014

When is it Better to Compare than to Score?

When eliciting judgements from humans for an unknown quantity, one often...

0 Nihar B. Shah, et al. ∙

Nihar B. Shah

Featured Co-authors