DeepAI AI Chat
Log In Sign Up

DAOS as HPC Storage, a view from Numerical Weather Prediction

by   Nicolau Manubens, et al.

Novel object storage solutions potentially address long-standing scalability issues with POSIX file systems, and Storage Class Memory (SCM) offers promising performance characteristics for data-intensive use cases. Intel's Distributed Asynchronous Object Store (DAOS) is an emerging high-performance object store which can leverage SCM and NVMe devices. It has been gaining traction after scoring top positions in the I/O 500 benchmark. Numerical Weather Prediction (NWP) simulations are sensitive to I/O performance and scaling, and their output resolution and diversity is expected to increase significantly in the near future. In this work, we present a preliminary assessment of DAOS in conjunction with SCM on a research HPC system and evaluate its potential use as HPC storage at a world-leading weather forecasting centre. We demonstrate DAOS can provide the required performance, with bandwidth scaling linearly with additional SCM nodes in most cases, although choices in configuration and application design can impact achievable bandwidth. We describe a new I/O benchmark and associated metrics that address object storage performance from application-derived workloads that can be utilised to explore real-world performance for this new class of storage systems


Performance Comparison of DAOS and Lustre for Object Data Storage Approaches

High-performance object stores are an emerging technology which offers a...

Evaluating Emerging CXL-enabled Memory Pooling for HPC Systems

Current HPC systems provide memory resources that are statically configu...

Exploring Scientific Application Performance Using Large Scale Object Storage

One of the major performance and scalability bottlenecks in large scient...

High Performance Parallel I/O and In-Situ Analysis in the WRF Model with ADIOS2

As the computing power of large-scale HPC clusters approaches the Exasca...

Analyzing I/O Performance of a Hierarchical HPC Storage System for Distributed Deep Learning

Today, deep learning is an essential technology for our life. To solve m...

accelerating wrf i/o performance with adios2 and network-based streaming

With the approach of Exascale computing power for large-scale High Perfo...

Object Storage, Persistent Memory, and Data Infrastructure for HPC Materials Informatics

Speculation is provided on how infrastructure choices fit into the mater...