Generative and reproducible benchmarks for comprehensive evaluation of machine learning classifiers

07/14/2021
by   Patryk Orzechowski, et al.
50

Understanding the strengths and weaknesses of machine learning (ML) algorithms is crucial for determine their scope of application. Here, we introduce the DIverse and GENerative ML Benchmark (DIGEN) - a collection of synthetic datasets for comprehensive, reproducible, and interpretable benchmarking of machine learning algorithms for classification of binary outcomes. The DIGEN resource consists of 40 mathematical functions which map continuous features to discrete endpoints for creating synthetic datasets. These 40 functions were discovered using a heuristic algorithm designed to maximize the diversity of performance among multiple popular machine learning algorithms thus providing a useful test suite for evaluating and comparing new methods. Access to the generative functions facilitates understanding of why a method performs poorly compared to other algorithms thus providing ideas for improvement. The resource with extensive documentation and analyses is open-source and available on GitHub.

READ FULL TEXT
research
12/06/2022

Benchmarking AutoML algorithms on a collection of binary problems

Automated machine learning (AutoML) algorithms have grown in popularity ...
research
10/21/2022

BEANS: The Benchmark of Animal Sounds

The use of machine learning (ML) based techniques has become increasingl...
research
07/15/2021

Data vs classifiers, who wins?

The classification experiments covered by machine learning (ML) are comp...
research
09/12/2023

MatSciML: A Broad, Multi-Task Benchmark for Solid-State Materials Modeling

We propose MatSci ML, a novel benchmark for modeling MATerials SCIence u...
research
07/29/2020

Decoding machine learning benchmarks

Despite the availability of benchmark machine learning (ML) repositories...
research
04/25/2019

Reviewing Data Access Patterns and Computational Redundancy for Machine Learning Algorithms

Machine learning (ML) is probably the first and foremost used technique ...
research
06/20/2023

Diverse Community Data for Benchmarking Data Privacy Algorithms

The Diverse Communities Data Excerpts are the core of a National Institu...

Please sign up or login with your details

Forgot password? Click here to reset