Design and Analysis of the NIPS 2016 Review Process

08/31/2017
by   Nihar B. Shah, et al.
0

Neural Information Processing Systems (NIPS) is a top-tier annual conference in machine learning. The 2016 edition of the conference comprised more than 2,400 paper submissions, 3,000 reviewers, and 8,000 attendees, representing a growth of nearly 40 over 100 report, we analyze several aspects of the data collected during the review process, including an experiment investigating the efficacy of collecting ordinal rankings from reviewers (vs. usual scores aka cardinal rankings). Our goal is to check the soundness of the review process we implemented and, in going so, provide insights that may be useful in the design of the review process of subsequent conferences. We introduce a number of metrics that could be used for monitoring improvements when new ideas are introduced.

READ FULL TEXT

page 10

page 12

page 17

research
04/05/2022

Integrating Rankings into Quantized Scores in Peer Review

In peer review, reviewers are usually asked to provide scores for the pa...
research
11/05/2018

National research assessment exercises: a comparison of peer review and bibliometrics rankings

Development of bibliometric techniques has reached such a level as to su...
research
01/07/2022

A Unified Statistical Learning Model for Rankings and Scores with Application to Grant Panel Review

Rankings and scores are two common data types used by judges to express ...
research
04/21/2023

The Isotonic Mechanism for Exponential Family Estimation

In 2023, the International Conference on Machine Learning (ICML) require...
research
06/05/2023

Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment

We present the NeurIPS 2021 consistency experiment, a larger-scale varia...
research
03/29/2017

Bringing Salary Transparency to the World: Computing Robust Compensation Insights via LinkedIn Salary

The recently launched LinkedIn Salary product has been designed with the...

Please sign up or login with your details

Forgot password? Click here to reset