Simulation as Experiment: An Empirical Critique of Simulation Research on Recommender Systems

by   Amy A. Winecoff, et al.

Simulation can enable the study of recommender system (RS) evolution while circumventing many of the issues of empirical longitudinal studies; simulations are comparatively easier to implement, are highly controlled, and pose no ethical risk to human participants. How simulation can best contribute to scientific insight about RS alongside qualitative and quantitative empirical approaches is an open question. Philosophers and researchers have long debated the epistemological nature of simulation compared to wholly theoretical or empirical methods. Simulation is often implicitly or explicitly conceptualized as occupying a middle ground between empirical and theoretical approaches, allowing researchers to realize the benefits of both. However, what is often ignored in such arguments is that without firm grounding in any single methodological tradition, simulation studies have no agreed upon scientific norms or standards, resulting in a patchwork of theoretical motivations, approaches, and implementations that are difficult to reconcile. In this position paper, we argue that simulation studies of RS are conceptually similar to empirical experimental approaches and therefore can be evaluated using the standards of empirical research methods. Using this empirical lens, we argue that the combination of high heterogeneity in approaches and low transparency in methods in simulation studies of RS has limited their interpretability, generalizability, and replicability. We contend that by adopting standards and practices common in empirical disciplines, simulation researchers can mitigate many of these weaknesses.



There are no comments yet.


page 1

page 2

page 3

page 4


The Ex-Ante View of Recommender System Design

Recommender systems (RS) are traditionally deployed in environments wher...

Simulations in Recommender Systems: An industry perspective

The construction of effective Recommender Systems (RS) is a complex proc...

Ethics of Food Recommender Applications

The recent unprecedented popularity of food recommender applications has...

T-RECS: A Simulation Tool to Study the Societal Impact of Recommender Systems

Simulation has emerged as a popular method to study the long-term societ...

Exposing SED Models And Snapshots Via VO Simulation Artefacts

The Virtual Observatory (VO) simulation standards, Simulation Data Model...

Towards Assessing Necessary Competence

We sketch a series of studies and experiments designed to provide empiri...

Using Camouflaged Cyber Simulations as a Model to Ensure Validity in Cybersecurity Experimentation

Experimental research methods describe standards to safeguard scientific...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.