DeepAI AI Chat
Log In Sign Up

DAOS as HPC Storage, a view from Numerical Weather Prediction

08/14/2022
by   Nicolau Manubens, et al.
ECMWF
0

Novel object storage solutions potentially address long-standing scalability issues with POSIX file systems, and Storage Class Memory (SCM) offers promising performance characteristics for data-intensive use cases. Intel's Distributed Asynchronous Object Store (DAOS) is an emerging high-performance object store which can leverage SCM and NVMe devices. It has been gaining traction after scoring top positions in the I/O 500 benchmark. Numerical Weather Prediction (NWP) simulations are sensitive to I/O performance and scaling, and their output resolution and diversity is expected to increase significantly in the near future. In this work, we present a preliminary assessment of DAOS in conjunction with SCM on a research HPC system and evaluate its potential use as HPC storage at a world-leading weather forecasting centre. We demonstrate DAOS can provide the required performance, with bandwidth scaling linearly with additional SCM nodes in most cases, although choices in configuration and application design can impact achievable bandwidth. We describe a new I/O benchmark and associated metrics that address object storage performance from application-derived workloads that can be utilised to explore real-world performance for this new class of storage systems

READ FULL TEXT
11/16/2022

Performance Comparison of DAOS and Lustre for Object Data Storage Approaches

High-performance object stores are an emerging technology which offers a...
11/04/2022

Evaluating Emerging CXL-enabled Memory Pooling for HPC Systems

Current HPC systems provide memory resources that are statically configu...
07/06/2018

Exploring Scientific Application Performance Using Large Scale Object Storage

One of the major performance and scalability bottlenecks in large scient...
01/20/2022

High Performance Parallel I/O and In-Situ Analysis in the WRF Model with ADIOS2

As the computing power of large-scale HPC clusters approaches the Exasca...
01/04/2023

Analyzing I/O Performance of a Hierarchical HPC Storage System for Distributed Deep Learning

Today, deep learning is an essential technology for our life. To solve m...
04/13/2023

accelerating wrf i/o performance with adios2 and network-based streaming

With the approach of Exascale computing power for large-scale High Perfo...
10/06/2022

Object Storage, Persistent Memory, and Data Infrastructure for HPC Materials Informatics

Speculation is provided on how infrastructure choices fit into the mater...