Enabling Reproducibility and Meta-learning Through a Lifelong Database of Experiments (LDE)

by   Jason Tsay, et al.

Artificial Intelligence (AI) development is inherently iterative and experimental. Over the course of normal development, especially with the advent of automated AI, hundreds or thousands of experiments are generated and are often lost or never examined again. There is a lost opportunity to document these experiments and learn from them at scale, but the complexity of tracking and reproducing these experiments is often prohibitive to data scientists. We present the Lifelong Database of Experiments (LDE) that automatically extracts and stores linked metadata from experiment artifacts and provides features to reproduce these artifacts and perform meta-learning across them. We store context from multiple stages of the AI development lifecycle including datasets, pipelines, how each is configured, and training runs with information about their runtime environment. The standardized nature of the stored metadata allows for querying and aggregation, especially in terms of ranking artifacts by performance metrics. We exhibit the capabilities of the LDE by reproducing an existing meta-learning study and storing the reproduced metadata in our system. Then, we perform two experiments on this metadata: 1) examining the reproducibility and variability of the performance metrics and 2) implementing a number of meta-learning algorithms on top of the data and examining how variability in experimental results impacts recommendation performance. The experimental results suggest significant variation in performance, especially depending on dataset configurations; this variation carries over when meta-learning is built on top of the results, with performance improving when using aggregated results. This suggests that a system that automatically collects and aggregates results such as the LDE not only assists in implementing meta-learning but may also improve its performance.


page 1

page 2

page 3

page 4


learn2learn: A Library for Meta-Learning Research

Meta-learning researchers face two fundamental issues in their empirical...

Towards Reproducible Empirical Research in Meta-Learning

Meta-learning is increasingly used to support the recommendation of mach...

Awesome-META+: Meta-Learning Research and Learning Platform

Artificial intelligence technology has already had a profound impact in ...

An Easy to Use Repository for Comparing and Improving Machine Learning Algorithm Usage

The results from most machine learning experiments are used for a specif...

Iterative Peptide Modeling With Active Learning And Meta-Learning

Often the development of novel materials is not amenable to high-through...

TIML: Task-Informed Meta-Learning for Agriculture

Labeled datasets for agriculture are extremely spatially imbalanced. Whe...

Building Containerized Environments for Reproducibility and Traceability of Scientific Workflows

Scientists rely on simulations to study natural phenomena. Trusting the ...

Please sign up or login with your details

Forgot password? Click here to reset