Python Workflows on HPC Systems

12/01/2020
by   Dominik Strassel, et al.
46

The recent successes and wide spread application of compute intensive machine learning and data analytics methods have been boosting the usage of the Python programming language on HPC systems. While Python provides many advantages for the users, it has not been designed with a focus on multi-user environments or parallel programming - making it quite challenging to maintain stable and secure Python workflows on a HPC system. In this paper, we analyze the key problems induced by the usage of Python on HPC clusters and sketch appropriate workarounds for efficiently maintaining multi-user Python software environments, securing and restricting resources of Python jobs and containing Python processes, while focusing on Deep Learning applications running on GPU clusters.

READ FULL TEXT
research
07/01/2021

Productivity, Portability, Performance: Data-Centric Python

Python has become the de facto language for scientific computing. Progra...
research
10/13/2020

Data Engineering for HPC with Python

Data engineering is becoming an increasingly important part of scientifi...
research
08/30/2022

The BioExcel methodology for developing dynamic, scalable, reliable and portable computational biomolecular workflows

Developing complex biomolecular workflows is not always straightforward....
research
07/03/2018

FluidFFT: common API (C++ and Python) for Fast Fourier Transform HPC libraries

The Python package fluidfft provides a common Python API for performing ...
research
03/22/2019

Active-Code Replacement in the OODIDA Data Analytics Platform

OODIDA (On-board/Off-board Distributed Data Analytics) is a platform for...
research
08/15/2023

Quantifying OpenMP: Statistical Insights into Usage and Adoption

In high-performance computing (HPC), the demand for efficient parallel p...
research
05/13/2021

Toward Real-time Analysis of Experimental Science Workloads on Geographically Distributed Supercomputers

Massive upgrades to science infrastructure are driving data velocities u...

Please sign up or login with your details

Forgot password? Click here to reset