Cross-replication Reliability – An Empirical Approach to Interpreting Inter-rater Reliability

06/11/2021

∙

by   Ka Wong, et al.

∙

0

∙

share

We present a new approach to interpreting IRR that is empirical and contextualized. It is based upon benchmarking IRR against baseline measures in a replication, one of which is a novel cross-replication reliability (xRR) measure based on Cohen's kappa. We call this approach the xRR framework. We opensource a replication dataset of 4 million human judgements of facial expressions and analyze it with the proposed framework. We argue this framework can be used to measure the quality of crowdsourced datasets.