DeepAI AI Chat
Log In Sign Up

Peer-review under review - A statistical study on proposal ranking at ESO. Part I: the pre-meeting phase

by   Ferdinando Patat, et al.
European Southern Observatory

Peer review is the most common mechanism in place for assessing requests for resources in a large variety of scientific disciplines. One of the strongest criticisms to this paradigm is the limited reproducibility of the process, especially at largely oversubscribed facilities. In this and in a subsequent paper we address this specific aspect in a quantitative way, through a statistical study on proposal ranking at the European Southern Observatory. For this purpose we analysed a sample of about 15000 proposals, submitted by more than 3000 Principal Investigators over 8 years. The proposals were reviewed by more than 500 referees, who assigned over 140000 grades in about 200 panel sessions. After providing a detailed analysis of the statistical properties of the sample, the paper presents an heuristic model based on these findings, which is then used to provide quantitative estimates of the reproducibility of the pre-meeting process. On average, about one third of the proposals ranked in the top quartile by one referee are ranked in the same quartile by any other referee of the panel. A similar value is observed for the bottom quartile. In the central quartiles, the agreement fractions are very marginally above the value expected for a fully aleatory process (25 between two panels composed by 6 referees is 55+/-5 the top and bottom quartiles. The corresponding fraction for the central quartiles is 33+/-5 obtained from boot-strapping the data for sub-panels composed by 3 referees, and fully consistent with the NIPS experiment. The post-meeting phase will be presented and discussed in a forthcoming paper.


page 6

page 7


Rethinking Resource Allocation in Science

US funding agencies alone distribute a yearly total of roughly 65B dolla...

Rethinking the Funding Line at the Swiss National Science Foundation: Bayesian Ranking and Lottery

Funding agencies rely on peer review and expert panels to select the res...

System-Generated Requests for Rewriting Proposals

We present an online deliberation system using mutual evaluation in orde...

Simulation Study on a New Peer Review Approach

The increasing volume of scientific publications and grant proposals has...

State-of-Art-Reviewing: A Radical Proposal to Improve Scientific Publication

Peer review forms the backbone of modern scientific manuscript evaluatio...