RawArray: A Simple, Fast, and Extensible Archival Format for Numeric Data

11/30/2021

∙

by   David S. Smith, et al.

∙

0

∙

share

Raw data sizes are growing and proliferating in scientific research, driven by the success of data-hungry computational methods, such as machine learning. The preponderance of proprietary and shoehorned data formats make computations slower and make it harder to reproduce research and to port methods to new platforms. Here we present the RawArray format: a simple, fast, and extensible format for archival storage of multidimensional numeric arrays on disk. The RawArray file format is a simple concatenation of a header array and a data array. The header comprises seven or more 64-bit unsigned integers. The array data can be anything. Arbitrary user metadata can be appended to an RawArray file if desired, for example to store measurement details, color palettes, or geolocation data. We present benchmarks showing a factor of 2–3× speedup over HDF5 for a range of array sizes and a speedup of up to 20× in reading the common deep learning datasets MNIST and CIFAR10.