Ambitious Data Science Can Be Painless

01/25/2019
by   Hatef Monajemi, et al.
0

Modern data science research can involve massive computational experimentation; an ambitious PhD in computational fields may do experiments consuming several million CPU hours. Traditional computing practices, in which researchers use laptops or shared campus-resident resources, are inadequate for experiments at the massive scale and varied scope that we now see in data science. On the other hand, modern cloud computing promises seemingly unlimited computational resources that can be custom configured, and seems to offer a powerful new venue for ambitious data-driven science. Exploiting the cloud fully, the amount of work that could be completed in a fixed amount of time can expand by several orders of magnitude. As potentially powerful as cloud-based experimentation may be in the abstract, it has not yet become a standard option for researchers in many academic disciplines. The prospect of actually conducting massive computational experiments in today's cloud systems confronts the potential user with daunting challenges. Leading considerations include: (i) the seeming complexity of today's cloud computing interface, (ii) the difficulty of executing an overwhelmingly large number of jobs, and (iii) the difficulty of monitoring and combining a massive collection of separate results. Starting a massive experiment `bare-handed' seems therefore highly problematic and prone to rapid `researcher burn out'. New software stacks are emerging that render massive cloud experiments relatively painless. Such stacks simplify experimentation by systematizing experiment definition, automating distribution and management of tasks, and allowing easy harvesting of results and documentation. In this article, we discuss several painless computing stacks that abstract away the difficulties of massive experimentation, thereby allowing a proliferation of ambitious experiments for scientific discovery.

READ FULL TEXT

page 4

page 7

page 9

research
10/28/2019

Reproducing Scientific Experiment with Cloud DevOps

The reproducibility of scientific experiment is vital for the advancemen...
research
01/20/2022

Diversifying the Genomic Data Science Research Community

Over the last 20 years, there has been an explosion of genomic data coll...
research
08/30/2022

Comparative Review of Cloud Computing Platforms for Data Science Workflows

With the advantages that cloud computing offers in terms of platform as ...
research
09/02/2021

Data science and Machine learning in the Clouds: A Perspective for the Future

As we are fast approaching the beginning of a paradigm shift in the fiel...
research
06/14/2023

Enforcing Data Geolocation Policies in Public Clouds using Trusted Computing

With the advancement in technology, Cloud computing always amazes the wo...
research
02/17/2021

Notebook articles: towards a transformative publishing experience in nonlinear science

Open Science, Reproducible Research, Findable, Accessible, Interoperable...

Please sign up or login with your details

Forgot password? Click here to reset