Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML

by   Shweta Narkar, et al.

Automated Machine Learning (AutoML) is a rapidly growing set of technologies that automate the model development pipeline by searching model space and generating candidate models. A critical, final step of AutoML is human selection of a final model from dozens of candidates. In current AutoML systems, selection is supported only by performance metrics. Prior work has shown that in practice, people evaluate ML models based on additional criteria, such as the way a model makes predictions. Comparison may happen at multiple levels, from types of errors, to feature importance, to how the model makes predictions of specific instances. We developed to support interactive model comparison for AutoML by integrating multiple Explainable AI (XAI) and visualization techniques. We conducted a user study in which we both evaluated the system and used it as a technology probe to understand how users perform model comparison in an AutoML system. We discuss design implications for utilizing XAI techniques for model comparison and supporting the unique needs of data scientists in comparing AutoML models.


page 1

page 2

page 3

page 4


explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning

We propose a framework for interactive and explainable machine learning ...

What Would You Ask the Machine Learning Model? Identification of User Needs for Model Explanations Based on Human-Model Conversations

Recently we see a rising number of methods in the field of eXplainable A...

EDAssistant: Supporting Exploratory Data Analysis in Computational Notebooks with In-Situ Code Search and Recommendation

Using computational notebooks (e.g., Jupyter Notebook), data scientists ...

Leveraging Rationales to Improve Human Task Performance

Machine learning (ML) systems across many application areas are increasi...

Towards Semantically Enhanced Data Understanding

In the field of machine learning, data understanding is the practice of ...

AI Total: Analyzing Security ML Models with Imperfect Data in Production

Development of new machine learning models is typically done on manually...