Serverless Supercomputing: High Performance Function as a Service for Science

by   Ryan Chard, et al.

Growing data volumes and velocities are driving exciting new methods across the sciences in which data analytics and machine learning are increasingly intertwined with research. These new methods require new approaches for scientific computing in which computation is mobile, so that, for example, it can occur near data, be triggered by events (e.g., arrival of new data), or be offloaded to specialized accelerators. They also require new design approaches in which monolithic applications can be decomposed into smaller components, that may in turn be executed separately and on the most efficient resources. To address these needs we propose funcX---a high-performance function-as-a-service (FaaS) platform that enables intuitive, flexible, efficient, scalable, and performant remote function execution on existing infrastructure including clouds, clusters, and supercomputers. It allows users to register and then execute Python functions without regard for the physical resource location, scheduler architecture, or virtualization technology on which the function is executed---an approach we refer to as "serverless supercomputing." We motivate the need for funcX in science, describe our prototype implementation, and demonstrate, via experiments on two supercomputers, that funcX can process millions of functions across more than 65000 concurrent workers. We also outline five scientific scenarios in which funcX has been deployed and highlight the benefits of funcX in these scenarios.


page 1

page 2

page 3

page 4


funcX: A Federated Function Serving Fabric for Science

Exploding data volumes and velocities, new computational methods and pla...

RFaaS: RDMA-Enabled FaaS Platform for Serverless High-Performance Computing

The rigid MPI programming model and batch scheduling dominate high-perfo...

Parsl: Pervasive Parallel Programming in Python

High-level programming languages such as Python are increasingly used to...

Molecular Dynamics Simulations on Cloud Computing and Machine Learning Platforms

Scientific computing applications have benefited greatly from high perfo...

EPIC: An Energy-Efficient, High-Performance GPGPU Computing Research Infrastructure

The pursuit of many research questions requires massive computational re...

Toward Real-time Analysis of Experimental Science Workloads on Geographically Distributed Supercomputers

Massive upgrades to science infrastructure are driving data velocities u...

The Trip to The Enterprise Gourmet Data Product Marketplace through a Self-service Data Platform

Data Analytics provides core business reporting needs in many software c...