DeepAI AI Chat
Log In Sign Up

Towards Reproducible Empirical Research in Meta-Learning

by   Adriano Rivolli, et al.
Universidade de São Paulo
Universidade do Porto
TU Eindhoven

Meta-learning is increasingly used to support the recommendation of machine learning algorithms and their configurations. Such recommendations are made based on meta-data, consisting of performance evaluations of algorithms on prior datasets, as well as characterizations of these datasets. These characterizations, also called meta-features, describe properties of the data which are predictive for the performance of machine learning algorithms trained on them. Unfortunately, despite being used in a large number of studies, meta-features are not uniformly described and computed, making many empirical studies irreproducible and hard to compare. This paper aims to remedy this by systematizing and standardizing data characterization measures used in meta-learning, and performing an in-depth analysis of their utility. Moreover, it presents MFE, a new tool for extracting meta-features from datasets and identify more subtle reproducibility issues in the literature, proposing guidelines for data characterization that strengthen reproducible empirical research in meta-learning.


learn2learn: A Library for Meta-Learning Research

Meta-learning researchers face two fundamental issues in their empirical...

Explaining the Performance of Multi-label Classification Methods with Data Set Properties

Meta learning generalizes the empirical experience with different learni...

Learning Abstract Task Representations

A proper form of data characterization can guide the process of learning...

Evaluating Meta-Feature Selection for the Algorithm Recommendation Problem

With the popularity of Machine Learning (ML) solutions, algorithms and d...

Enabling Reproducibility and Meta-learning Through a Lifelong Database of Experiments (LDE)

Artificial Intelligence (AI) development is inherently iterative and exp...

An Introduction to Advanced Machine Learning : Meta Learning Algorithms, Applications and Promises

In [1, 2], we have explored the theoretical aspects of feature extractio...

Meta-learning Control Variates: Variance Reduction with Limited Data

Control variates can be a powerful tool to reduce the variance of Monte ...