Equal Affection or Random Selection: the Quality of Subjective Feedback from a Group Perspective

02/24/2021
by   Jiale Chen, et al.
0

In the setting where a group of agents is asked a single subjective multi-choice question (e.g. which one do you prefer? cat or dog?), we are interested in evaluating the quality of the collected feedback. However, the collected statistics are not sufficient to reflect how informative the feedback is since fully informative feedback (equal affection of the choices) and fully uninformative feedback (random selection) have the same uniform statistics. Here we distinguish the above two scenarios by additionally asking for respondents' predictions about others' choices. We assume that informative respondents' predictions strongly depend on their own choices while uninformative respondents' do not. With this assumption, we propose a new definition for uninformative feedback and correspondingly design a family of evaluation metrics, called f-variety, for group-level feedback which can 1) distinguish informative feedback and uninformative feedback (separation) even if their statistics are both uniform and 2) decrease as the ratio of uninformative respondents increases (monotonicity). We validate our approach both theoretically and numerically. Moreover, we conduct two real-world case studies about 1) comparisons about athletes and 2) comparisons about stand-up comedians to show the superiority of our approach.

READ FULL TEXT

page 6

page 17

research
08/23/2022

The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types

When inferring reward functions from human behavior (be it demonstration...
research
06/23/2023

System-Level Natural Language Feedback

Natural language (NL) feedback contains rich information about the user ...
research
01/15/2021

Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback

Crowdsourced design feedback systems are emerging resources for getting ...
research
11/06/2019

Nonverbal Robot Feedback for Human Teachers

Robots can learn preferences from human demonstrations, but their succes...
research
06/07/2022

An Exploratory Analysis of Feedback Types Used in Online Coding Exercises

Online coding environments can help support computing students gain prog...
research
10/30/2018

Designing Informative Rating Systems for Online Platforms: Evidence from Two Experiments

Platforms critically rely on rating systems to learn the quality of mark...

Please sign up or login with your details

Forgot password? Click here to reset