Selecting applicants based on multiple ratings: Using binary classification framework as an alternative to inter-rater reliability

07/19/2022
by   František Bartoš, et al.
0

Inter-rater reliability (IRR) has been the prevalent quality and precision measure in ratings from multiple raters. However, applicant selection procedures based on ratings from multiple raters usually result in a binary outcome. This final outcome is not considered in IRR, which instead focuses on the ratings of the individual subjects or objects. In this work, we outline how to transform the selection procedures into a binary classification framework and develop a quantile approximation which connects a measurement model for the ratings with the binary classification framework. The quantile approximation allows us to estimate the probability of correctly selecting the best applicants and assess error probabilities when evaluating the quality of selection procedures using ratings from multiple raters. We draw connections between the inter-rater reliability and the binary classification metrics, showing that binary classification metrics depend solely on the IRR coefficient and proportion of selected applicants. We assess the performance of the quantile approximation in a simulation study and apply it in an example comparing the reliability of multiple grant peer review selection procedures.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/24/2022

k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations

Since the inception of crowdsourcing, aggregation has been a common stra...
research
08/03/2023

Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratings

This study investigates the consistency of feedback ratings generated by...
research
07/04/2021

Adaptive calibration for binary classification

This note proposes a way of making probability forecasting rules less se...
research
01/16/2021

Towards Searching Efficient and Accurate Neural Network Architectures in Binary Classification Problems

In recent years, deep neural networks have had great success in machine ...
research
07/24/2021

A Model-Agnostic Algorithm for Bayes Error Determination in Binary Classification

This paper presents the intrinsic limit determination algorithm (ILD Alg...
research
06/27/2012

A Binary Classification Framework for Two-Stage Multiple Kernel Learning

With the advent of kernel methods, automating the task of specifying a s...
research
05/07/2015

Optimal Decision-Theoretic Classification Using Non-Decomposable Performance Metrics

We provide a general theoretical analysis of expected out-of-sample util...

Please sign up or login with your details

Forgot password? Click here to reset