An Empirical Study on the Usage of Automated Machine Learning Tools

by   Forough Majidi, et al.

The popularity of automated machine learning (AutoML) tools in different domains has increased over the past few years. Machine learning (ML) practitioners use AutoML tools to automate and optimize the process of feature engineering, model training, and hyperparameter optimization and so on. Recent work performed qualitative studies on practitioners' experiences of using AutoML tools and compared different AutoML tools based on their performance and provided features, but none of the existing work studied the practices of using AutoML tools in real-world projects at a large scale. Therefore, we conducted an empirical study to understand how ML practitioners use AutoML tools in their projects. To this end, we examined the top 10 most used AutoML tools and their respective usages in a large number of open-source project repositories hosted on GitHub. The results of our study show 1) which AutoML tools are mostly used by ML practitioners and 2) the characteristics of the repositories that use these AutoML tools. Also, we identified the purpose of using AutoML tools (e.g. model parameter sampling, search space management, model evaluation/error-analysis, Data/ feature transformation, and data labeling) and the stages of the ML pipeline (e.g. feature engineering) where AutoML tools are used. Finally, we report how often AutoML tools are used together in the same source code files. We hope our results can help ML practitioners learn about different AutoML tools and their usages, so that they can pick the right tool for their purposes. Besides, AutoML tool developers can benefit from our findings to gain insight into the usages of their tools and improve their tools to better fit the users' usages and needs.


page 1

page 9


An Empirical Study of Bots in Software Development – Characteristics and Challenges from a Practitioner's Perspective

Software engineering bots - automated tools that handle tedious tasks - ...

Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Tools

There has been considerable growth and interest in industrial applicatio...

AQuA: A Benchmarking Tool for Label Quality Assessment

Machine learning (ML) models are only as good as the data they are train...

Analysing Wideband Absorbance Immittance in Normal and Ears with Otitis Media with Effusion Using Machine Learning

Wideband Absorbance Immittance (WAI) has been available for more than a ...

Azimuth: Systematic Error Analysis for Text Classification

We present Azimuth, an open-source and easy-to-use tool to perform error...

A Qualitative and Quantitative Analysis of Container Engines

Containerization is a virtualization technique that allows one to create...

Angler: Helping Machine Translation Practitioners Prioritize Model Improvements

Machine learning (ML) models can fail in unexpected ways in the real wor...

Please sign up or login with your details

Forgot password? Click here to reset