Progressive Data Science: Potential and Challenges

by   Cagatay Turkay, et al.

Data science requires time-consuming iterative manual activities. In particular, activities such as data selection, preprocessing, transformation, and mining, highly depend on iterative trial-and-error processes that could be sped up significantly by providing quick feedback on the impact of changes. The idea of progressive data science is to compute the results of changes in a progressive manner, returning a first approximation of results quickly and allow iterative refinements until converging to a final result. Enabling the user to interact with the intermediate results allows an early detection of erroneous or suboptimal choices, the guided definition of modifications to the pipeline and their quick assessment. In this paper, we discuss the progressiveness challenges arising in different steps of the data science pipeline. We describe how changes in each step of the pipeline impact the subsequent steps and outline why progressive data science will help to make the process more effective. Computing progressive approximations of outcomes resulting from changes creates numerous research challenges, especially if the changes are made in the early steps of the pipeline. We discuss these challenges and outline first steps towards progressiveness, which, we argue, will ultimately help to significantly speed-up the overall data science process.


page 1

page 2

page 3

page 4


Perspective on Data Science

The field of data science currently enjoys a broad definition that inclu...

Teaching Visual Accessibility in Introductory Data Science Classes with Multi-Modal Data Representations

Although there are various ways to represent data patterns and models, v...

How Do Data Science Workers Communicate Intermediate Results?

Data science workers increasingly collaborate on large-scale projects be...

Linked Data Science Powered by Knowledge Graphs

In recent years, we have witnessed a growing interest in data science no...

Problem Formulation and Fairness

Formulating data science problems is an uncertain and difficult process....

Towards "all-inclusive" Data Preparation to ensure Data Quality

Data preparation, especially data cleaning, is very important to ensure ...

A Novel Gradient Methodology with Economical Objective Function Evaluations for Data Science Applications

Gradient methods are experiencing a growth in methodological and theoret...

Please sign up or login with your details

Forgot password? Click here to reset