A Serverless Tool for Platform Agnostic Computational Experiment Management

09/02/2018
by   Gregory Kiar, et al.
0

Neuroscience has been carried into the domain of big data and high performance computing (HPC) on the backs of initiatives in data collection and an increasingly compute-intensive tools. While managing HPC experiments requires considerable technical acumen, platforms and standards have been developed to ease this burden on scientists. While web-portals make resources widely accessible, data organizations such as the Brain Imaging Data Structure and tool description languages such as Boutiques provide researchers with a foothold to tackle these problems using their own datasets, pipelines, and environments. While these standards lower the barrier to adoption of HPC and cloud systems for neuroscience applications, they still require the consolidation of disparate domain-specific knowledge. We present Clowdr, a lightweight tool to launch experiments on HPC systems and clouds, record rich execution records, and enable the accessible sharing of experimental summaries and results. Clowdr uniquely sits between web platforms and bare-metal applications for experiment management by preserving the flexibility of do-it-yourself solutions while providing a low barrier for developing, deploying and disseminating neuroscientific analysis.

READ FULL TEXT
research
05/16/2018

A Software-Defined QoS Provisioning Framework for HPC Applications

With the emergence of large-scale data-intensive high-performance applic...
research
02/11/2019

Scaling Big Data Platform for Big Data Pipeline

Monitoring and Managing High Performance Computing (HPC) systems and env...
research
12/16/2020

An Integrated Platform for Collaborative Data Analytics

While collaboration among data scientists is a key to organizational pro...
research
06/26/2023

A Comparison of Neuroelectrophysiology Databases

As data sharing has become more prevalent, three pillars - archives, sta...
research
08/30/2022

The BioExcel methodology for developing dynamic, scalable, reliable and portable computational biomolecular workflows

Developing complex biomolecular workflows is not always straightforward....
research
06/09/2015

Reproducible and User-Controlled Software Environments in HPC with Guix

Support teams of high-performance computing (HPC) systems often find the...
research
08/18/2023

Scope is all you need: Transforming LLMs for HPC Code

With easier access to powerful compute resources, there is a growing tre...

Please sign up or login with your details

Forgot password? Click here to reset