When is it Better to Compare than to Score?

06/25/2014
by   Nihar B. Shah, et al.
0

When eliciting judgements from humans for an unknown quantity, one often has the choice of making direct-scoring (cardinal) or comparative (ordinal) measurements. In this paper we study the relative merits of either choice, providing empirical and theoretical guidelines for the selection of a measurement scheme. We provide empirical evidence based on experiments on Amazon Mechanical Turk that in a variety of tasks, (pairwise-comparative) ordinal measurements have lower per sample noise and are typically faster to elicit than cardinal ones. Ordinal measurements however typically provide less information. We then consider the popular Thurstone and Bradley-Terry-Luce (BTL) models for ordinal measurements and characterize the minimax error rates for estimating the unknown quantity. We compare these minimax error rates to those under cardinal measurement models and quantify for what noise levels ordinal measurements are better. Finally, we revisit the data collected from our experiments and show that fitting these models confirms this prediction: for tasks where the noise in ordinal measurements is sufficiently low, the ordinal approach results in smaller errors in the estimation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/06/2015

Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence

Data in the form of pairwise comparisons arises in many domains, includi...
research
02/23/2016

Lens depth function and k-relative neighborhood graph: versatile tools for ordinal data analysis

In recent years it has become popular to study machine learning problems...
research
03/04/2018

Complexity and (un)decidability of fragments of 〈 ω^ω^λ; ×〉

We specify the frontier of decidability for fragments of the first-order...
research
06/22/2016

Finite Sample Prediction and Recovery Bounds for Ordinal Embedding

The goal of ordinal embedding is to represent items as points in a low-d...
research
04/30/2013

Inferring ground truth from multi-annotator ordinal data: a probabilistic approach

A popular approach for large scale data annotation tasks is crowdsourcin...
research
06/01/2020

An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results

In Ordinal Classification tasks, items have to be assigned to classes th...
research
05/22/2023

Ordinal Outcome State-Space Models for Intensive Longitudinal Data

Intensive longitudinal (IL) data are increasingly prevalent in psycholog...

Please sign up or login with your details

Forgot password? Click here to reset