Choosing How to Choose Papers
share
It is common to see a handful of reviewers reject a highly novel paper, because they view, say, extensive experiments as far more important than novelty, whereas the community as a whole would have embraced the paper. More generally, the disparate mapping of criteria scores to final recommendations by different reviewers is a major source of inconsistency in peer review. In this paper we present a framework --- based on L(p,q)-norm empirical risk minimization --- for learning the community's aggregate mapping. We draw on computational social choice to identify desirable values of p and q; specifically, we characterize p=q=1 as the only choice that satisfies three natural axiomatic properties. Finally, we implement and apply our approach to reviews from IJCAI 2017.
READ FULL TEXT