Designing Informative Rating Systems for Online Platforms: Evidence from Two Experiments

10/30/2018
by   Nikhil Garg, et al.
0

Platforms critically rely on rating systems to learn the quality of market participants. In practice, however, these ratings are often highly inflated, drastically reducing the signal available to distinguish quality. We consider two questions: First, can rating systems better discriminate quality by altering the meaning and relative importance of the levels in the rating system? And second, if so, how should the platform optimize these choices in the design of the rating system? We first analyze the results of a randomized controlled trial on an online labor market in which an additional question was added to the feedback form. Between treatment conditions, we vary the question phrasing and answer choices. We further run an experiment on Amazon Mechanical Turk with similar structure, to confirm the labor market findings. Our tests reveal that current inflationary norms can in fact be countered by re-anchoring the meaning of the levels of the rating system. In particular, scales that are positive-skewed and provide specific interpretations for what each label means yield rating distributions that are much more informative about quality. Second, we develop a theoretical framework to optimize the design of a rating system by choosing answer labels and their numeric interpretations in a manner that maximizes the rate of convergence to the true underlying quality distribution. Finally, we run simulations with an empirically calibrated model and use these to study the implications for optimal rating system design. Our simulations demonstrate that our modeling and optimization approach can substantially improve the quality of information obtained over baseline designs. Overall, our study illustrates that rating systems that are informative in practice can be designed, and demonstrates how to design them in a principled manner.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/18/2018

Designing Optimal Binary Rating Systems

Modern online platforms rely on effective rating systems to learn about ...
research
07/23/2023

Interface Design to Mitigate Inflation in Recommender Systems

Recommendation systems rely on user-provided data to learn about item qu...
research
01/26/2021

RewardRating: A Mechanism Design Approach to Improve Rating Systems

Nowadays, rating systems play a crucial role in the attraction of custom...
research
01/26/2021

Mining the Stars: Learning Quality Ratings with User-facing Explanations for Vacation Rentals

Online Travel Platforms are virtual two-sided marketplaces where guests ...
research
02/19/2018

When Sheep Shop: Measuring Herding Effects in Product Ratings with Natural Experiments

As online shopping becomes ever more prevalent, customers rely increasin...
research
02/13/2020

Zero-Rating and Net Neutrality: Who Wins, Who Loses?

An objective of network neutrality is that the design of regulations for...
research
02/24/2021

Equal Affection or Random Selection: the Quality of Subjective Feedback from a Group Perspective

In the setting where a group of agents is asked a single subjective mult...

Please sign up or login with your details

Forgot password? Click here to reset