RNTuple performance: Status and Outlook

04/07/2022
by   Javier López-Gómez, et al.
0

Upcoming HEP experiments, e.g. at the HL-LHC, are expected to increase the volume of generated data by at least one order of magnitude. In order to retain the ability to analyze the influx of data, full exploitation of modern storage hardware and systems, such as low-latency high-bandwidth NVMe devices and distributed object stores, becomes critical. To this end, the ROOT RNTuple I/O subsystem has been designed to address performance bottlenecks and shortcomings of ROOT's current state of the art TTree I/O subsystem. RNTuple provides a backwards-incompatible redesign of the TTree binary format and access API that evolves the ROOT event data I/O for the challenges of the upcoming decades. It focuses on a compact data format, on performance engineering for modern storage hardware, for instance through making parallel and asynchronous I/O calls by default, and on robust interfaces that are easy to use correctly. In this contribution, we evaluate the RNTuple performance for typical HEP analysis tasks. We compare the throughput delivered by RNTuple to popular I/O libraries outside HEP, such as HDF5 and Apache Parquet. We demonstrate the advantages of RNTuple for HEP analysis workflows and provide an outlook on the road to its use in production.

READ FULL TEXT

Authors

07/15/2021

Exploring Object Stores for High-Energy Physics Data Storage

Over the last two decades, ROOT TTree has been used for storing over one...
03/17/2020

Evolution of the ROOT Tree I/O

The ROOT TTree data format encodes hundreds of petabytes of High Energy ...
07/03/2020

Mapping Datasets to Object Storage System

Access libraries such as ROOT and HDF5 allow users to interact with data...
04/20/2020

Vilamb: Low Overhead Asynchronous Redundancy for Direct Access NVM

Vilamb provides efficient asynchronous systemredundancy for direct acces...
11/07/2017

Optimizing ROOT IO For Analysis

The ROOT I/O (RIO) subsystem is foundational to most HEP experiments - i...
04/10/2018

Increasing Parallelism in the ROOT I/O Subsystem

When processing large amounts of data, the rate at which reading and wri...
06/11/2019

Speeding HEP Analysis with ROOT Bulk I/O

Distinct HEP workflows have distinct I/O needs; while ROOT I/O excels at...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.