SysML'19 demo: customizable and reusable Collective Knowledge pipelines to automate and reproduce machine learning experiments

03/31/2019
by   Grigori Fursin, et al.
0

Reproducing, comparing and reusing results from machine learning and systems papers is a very tedious, ad hoc and time-consuming process. I will demonstrate how to automate this process using open-source, portable, customizable and CLI-based Collective Knowledge workflows and pipelines developed by the community. I will help participants run several real-world non-virtualized CK workflows from the SysML'19 conference, companies (General Motors, Arm) and MLPerf benchmark to automate benchmarking and co-design of efficient software/hardware stacks for machine learning workloads. I hope that our approach will help authors reduce their effort when sharing reusable and extensible research artifacts while enabling artifact evaluators to automatically validate experimental results from published papers in a standard and portable way.

READ FULL TEXT

page 1

page 2

research
11/02/2020

Collective Knowledge: organizing research projects as a database of reusable components and portable workflows with common APIs

This article provides the motivation and overview of the Collective Know...
research
04/29/2019

Challenges and Pitfalls of Reproducing Machine Learning Artifacts

An increasingly complex and diverse collection of Machine Learning(ML) m...
research
04/29/2019

Challenges and Pitfalls of Machine Learning Evaluation and Benchmarking

An increasingly complex and diverse collection of Machine Learning (ML) ...
research
01/19/2018

Introducing ReQuEST: an Open Platform for Reproducible and Quality-Efficient Systems-ML Tournaments

Co-designing efficient machine learning based systems across the whole h...
research
01/19/2018

A Collective Knowledge workflow for collaborative research into multi-objective autotuning and machine learning techniques

Developing efficient software and hardware has never been harder whether...
research
01/19/2022

Automation of Citation Screening for Systematic Literature Reviews using Neural Networks: A Replicability Study

In the process of Systematic Literature Review, citation screening is es...

Please sign up or login with your details

Forgot password? Click here to reset