DeepAI
Log In Sign Up

Context-aware Execution Migration Tool for Data Science Jupyter Notebooks on Hybrid Clouds

07/01/2021
by   Renato L. F. Cunha, et al.
10

Interactive computing notebooks, such as Jupyter notebooks, have become a popular tool for developing and improving data-driven models. Such notebooks tend to be executed either in the user's own machine or in a cloud environment, having drawbacks and benefits in both approaches. This paper presents a solution developed as a Jupyter extension that automatically selects which cells, as well as in which scenarios, such cells should be migrated to a more suitable platform for execution. We describe how we reduce the execution state of the notebook to decrease migration time and we explore the knowledge of user interactivity patterns with the notebook to determine which blocks of cells should be migrated. Using notebooks from Earth science (remote sensing), image recognition, and hand written digit identification (machine learning), our experiments show notebook state reductions of up to 55x and migration decisions leading to performance gains of up to 3.25x when the user interactivity with the notebook is taken into consideration.

READ FULL TEXT

page 2

page 7

page 8

01/14/2021

Checkpoint, Restore, and Live Migration for Science Platforms

We demonstrate a fully functional implementation of (per-user) checkpoin...
04/23/2019

Migration patterns under different scenarios of sea level rise

We propose a framework to examine future migration patterns of people un...
12/04/2020

Supporting Round-Trip Data Migration for Web APIs: A Henshin Solution

We present a solution to the Round-Trip Migration case of the Transforma...
12/13/2020

Fine-Grained Lineage for Safer Notebook Interactions

Computational notebooks have emerged as the platform of choice for data ...
02/16/2022

Knowledge Management for Cloud Computing Field

Migration legacy systems to cloud platforms is a knowledge intensive pro...
03/03/2020

Data Migration using Datalog Program Synthesis

This paper presents a new technique for migrating data between different...