Ratings to Ranking: Preference Elicitation and Aggregation for Student Peer Assessment
share
Voters are usually asked to either rank or rate alternatives. However, reducing their task to just this or the other conceals essential information about their preferences. We propose a model consisting of two parts. First, we present an algorithm that elicits voter preferences: voters are asked to evaluate alternatives and respond to pairwise comparison queries when necessary. Secondly, we present a protocol for aggregating the voters' preferences into a single ranking. We implemented a system, R2R, that collects and aggregates user preferences. The system was deployed in a user study on student peer reviews, and the data obtained were used to evaluate R2R aggregation against state-of-the-art methods. Experiments conclude that when the number of voters is small (up to 10-15), R2R outputs a ranked list of alternatives with fewer ties between alternatives. Furthermore, R2R elicitation reduces the communication load on the voters by ∼ 70%.
READ FULL TEXT