Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings
share
Cardinal scores (numeric ratings) collected from people are well known to suffer from miscalibrations. A popular approach to address this issue is to assume simplistic models of miscalibration (such as linear biases) to de-bias the scores. This approach, however, often fares poorly because people's miscalibrations are typically far more complex and not well understood. In the absence of simplifying assumptions on the miscalibration, it is widely believed that the only useful information in the cardinal scores is the induced ranking. In this paper, inspired by the framework of Stein's shrinkage and empirical Bayes, we contest this widespread belief. Specifically, we consider cardinal scores with arbitrary (or even adversarially chosen) miscalibrations that is only required to be consistent with the induced ranking. We design estimators that despite making no assumptions on the miscalibration, surprisingly, strictly and uniformly outperform all possible estimators that rely on only the ranking. Our estimators are flexible in that they can be used as a plug-in for a variety of applications. Our results thus provide novel insights in the eternal debate between cardinal and ordinal data.
READ FULL TEXT