Two-Sample Testing on Ranked Preference Data and the Role of Modeling Assumptions

06/21/2020
by   Charvi Rastogi, et al.
0

A number of applications require two-sample testing on ranked preference data. For instance, in crowdsourcing, there is a long-standing question of whether pairwise comparison data provided by people is distributed similar to ratings-converted-to-comparisons. Other examples include sports data analysis and peer grading. In this paper, we design two-sample tests for pairwise comparison data and ranking data. For our two-sample test for pairwise comparison data, we establish an upper bound on the sample complexity required to correctly distinguish between the distributions of the two sets of samples. Our test requires essentially no assumptions on the distributions. We then prove complementary lower bounds showing that our results are tight (in the minimax sense) up to constant factors. We investigate the role of modeling assumptions by proving lower bounds for a range of pairwise comparison models (WST, MST,SST, parameter-based such as BTL and Thurstone). We also provide testing algorithms and associated sample complexity bounds for the problem of two-sample testing with partial (or total) ranking data.Furthermore, we empirically evaluate our results via extensive simulations as well as two real-world datasets consisting of pairwise comparisons. By applying our two-sample test on real-world pairwise comparison data, we conclude that ratings and rankings provided by people are indeed distributed differently. On the other hand, our test recognizes no significant difference in the relative performance of European football teams across two seasons. Finally, we apply our two-sample test on a real-world partial and total ranking dataset and find a statistically significant difference in Sushi preferences across demographic divisions based on gender, age and region of residence.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/08/2018

PAC Ranking from Pairwise and Listwise Queries: Lower Bounds and Upper Bounds

This paper explores the adaptively (active) PAC (probably approximately ...
research
09/07/2019

On Sample Complexity Upper and Lower Bounds for Exact Ranking from Noisy Comparisons

This paper studies the problem of finding the exact ranking from noisy c...
research
08/11/2018

Ranking with Features: Algorithm and A Graph Theoretic Analysis

We consider the problem of ranking a set of items from pairwise comparis...
research
05/04/2021

On the Sample Complexity of Rank Regression from Pairwise Comparisons

We consider a rank regression setting, in which a dataset of N samples w...
research
09/12/2022

Is Synthetic Dataset Reliable for Benchmarking Generalizable Person Re-Identification?

Recent studies show that models trained on synthetic datasets are able t...
research
06/03/2019

Optimal Learning of Mallows Block Model

The Mallows model, introduced in the seminal paper of Mallows 1957, is o...
research
12/28/2022

Ratings to Ranking: Preference Elicitation and Aggregation for Student Peer Assessment

Voters are usually asked to either rank or rate alternatives. However, r...

Please sign up or login with your details

Forgot password? Click here to reset