Checkpoint, Restore, and Live Migration for Science Platforms

01/14/2021
by   Mario Jurić, et al.
0

We demonstrate a fully functional implementation of (per-user) checkpoint, restore, and live migration capabilities for JupyterHub platforms. Checkpointing – the ability to freeze and suspend to disk the running state (contents of memory, registers, open files, etc.) of a set of processes – enables the system to snapshot a user's Jupyter session to permanent storage. The restore functionality brings a checkpointed session back to a running state, to continue where it left off at a later time and potentially on a different machine. Finally, live migration enables moving running Jupyter notebook servers between different machines, transparent to the analysis code and w/o disconnecting the user. Our implementation of these capabilities works at the system level, with few limitations, and typical checkpoint/restore times of O(10s) with a pathway to O(1s) live migrations. It opens a myriad of interesting use cases, especially for cloud-based deployments: from checkpointing idle sessions w/o interruption of the user's work (achieving cost reductions of 4x or more), execution on spot instances w. transparent migration on eviction (with additional cost reductions up to 3x), to automated migration of workloads to ideally suited instances (e.g. moving an analysis to a machine with more or less RAM or cores based on observed resource utilization). The capabilities we demonstrate can make science platforms fully elastic while retaining excellent user experience.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

03/08/2020

Survey Study of Virtual Machine Migration Techniques in Cloud Computing

Migration of virtual machine is one of the most important features in vi...
09/15/2020

MigrOS: Transparent Operating Systems Live Migration Support for Containerised RDMA-applications

Major data centre providers are introducing RDMA-based networks for thei...
07/01/2021

Context-aware Execution Migration Tool for Data Science Jupyter Notebooks on Hybrid Clouds

Interactive computing notebooks, such as Jupyter notebooks, have become ...
01/12/2013

Dynamic Transparent General Purpose Process Migration For Linux

Process migration refers to the act of transferring a process in the mid...
12/02/2017

Toward Reliable and Rapid Elasticity for Streaming Dataflows on Clouds

The pervasive availability of streaming data is driving interest in dist...
12/27/2012

LNOS - Live Network Operating System

Operating Systems exists since existence of computers, and have been evo...
12/02/2021

Simplifying heterogeneous migration between x86 and ARM machines

Heterogeneous computing is the strategy of deploying multiple types of p...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.