Elliot: a Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation

03/03/2021
by   Vito Walter Anelli, et al.
0

Recommender Systems have shown to be an effective way to alleviate the over-choice problem and provide accurate and tailored recommendations. However, the impressive number of proposed recommendation algorithms, splitting strategies, evaluation protocols, metrics, and tasks, has made rigorous experimental evaluation particularly challenging. Puzzled and frustrated by the continuous recreation of appropriate evaluation benchmarks, experimental pipelines, hyperparameter optimization, and evaluation procedures, we have developed an exhaustive framework to address such needs. Elliot is a comprehensive recommendation framework that aims to run and reproduce an entire experimental pipeline by processing a simple configuration file. The framework loads, filters, and splits the data considering a vast set of strategies (13 splitting methods and 8 filtering approaches, from temporal training-test splitting to nested K-folds Cross-Validation). Elliot optimizes hyperparameters (51 strategies) for several recommendation algorithms (50), selects the best models, compares them with the baselines providing intra-model statistics, computes metrics (36) spanning from accuracy to beyond-accuracy, bias, and fairness, and conducts statistical analysis (Wilcoxon and Paired t-test). The aim is to provide the researchers with a tool to ease (and make them reproducible) all the experimental evaluation phases, from data reading to results collection. Elliot is available on GitHub (https://github.com/sisinflab/elliot).

READ FULL TEXT
research
07/26/2020

Exploring Data Splitting Strategies for the Evaluation of Recommendation Models

Effective methodologies for evaluating recommender systems are critical,...
research
11/03/2020

RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms

In recent years, there are a large number of recommendation algorithms p...
research
06/22/2022

DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation

Recently, one critical issue looms large in the field of recommender sys...
research
09/30/2020

MARS-Gym: A Gym framework to model, train, and evaluate Recommender Systems for Marketplaces

Recommender Systems are especially challenging for marketplaces since th...
research
01/18/2022

Recommendation Unlearning

Recommender systems provide essential web services by learning users' pe...
research
02/09/2023

A Comprehensive Survey on Multimodal Recommender Systems: Taxonomy, Evaluation, and Future Directions

Recommendation systems have become popular and effective tools to help u...

Please sign up or login with your details

Forgot password? Click here to reset