On the discriminative power of Hyper-parameters in Cross-Validation and how to choose them

09/05/2019
by   Vito Walter Anelli, et al.
0

Hyper-parameters tuning is a crucial task to make a model perform at its best. However, despite the well-established methodologies, some aspects of the tuning remain unexplored. As an example, it may affect not just accuracy but also novelty as well as it may depend on the adopted dataset. Moreover, sometimes it could be sufficient to concentrate on a single parameter only (or a few of them) instead of their overall set. In this paper we report on our investigation on hyper-parameters tuning by performing an extensive 10-Folds Cross-Validation on MovieLens and Amazon Movies for three well-known baselines: User-kNN, Item-kNN, BPR-MF. We adopted a grid search strategy considering approximately 15 values for each parameter, and we then evaluated each combination of parameters in terms of accuracy and novelty. We investigated the discriminative power of nDCG, Precision, Recall, MRR, EFD, EPC, and, finally, we analyzed the role of parameters on model evaluation for Cross-Validation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/16/2016

Assessing and tuning brain decoders: cross-validation, caveats, and guidelines

Decoding, ie prediction from brain images or signals, calls for empirica...
research
12/31/2021

Improving Baselines in the Wild

We share our experience with the recently released WILDS benchmark, a co...
research
05/28/2022

Rethinking the Setting of Semi-supervised Learning on Graphs

We argue that the present setting of semisupervised learning on graphs m...
research
05/27/2014

Futility Analysis in the Cross-Validation of Machine Learning Models

Many machine learning models have important structural tuning parameters...
research
08/22/2019

Efficient Cross-Validation of Echo State Networks

Echo State Networks (ESNs) are known for their fast and precise one-shot...
research
03/28/2019

An analysis of the cost of hyper-parameter selection via split-sample validation, with applications to penalized regression

In the regression setting, given a set of hyper-parameters, a model-esti...
research
06/19/2020

Efficient implementations of echo state network cross-validation

Background/introduction: Cross-validation is still uncommon in time seri...

Please sign up or login with your details

Forgot password? Click here to reset