[Paper Review] A Comparison of HDF5, Zarr, and netCDF4 in Performing Common I/O Operations

Sriniket Ambatipudi, Suren Byna|arXiv (Cornell University)|Jul 19, 2022

Advanced Data Storage Technologies4 citations

TL;DR

This study benchmarks HDF5, netCDF4, and Zarr for common I/O operations—create, open, write, read, and close—using synthetic datasets in Python. Results show HDF5 is fastest for read/write operations, netCDF4 excels in create/open times, and Zarr trails slightly behind HDF5 across most operations, with performance heavily influenced by API overhead and caching in small-data scenarios.

ABSTRACT

Scientific data is often stored in files because of the simplicity they provide in managing, transferring, and sharing data. These files are typically structured in a specific arrangement and contain metadata to understand the structure the data is stored in. There are numerous file formats in use in various scientific domains that provide abstractions for storing and retrieving data. With the abundance of file formats aiming to store large amounts of scientific data quickly and easily, a question that arises is, "Which scientific file format is best suited for a general use case?" In this study, we compiled a set of benchmarks for common file operations, i.e., create, open, read, write, and close, and used the results of these benchmarks to compare three popular formats: HDF5, netCDF4, and Zarr.

Motivation & Objective

• To objectively compare the I/O performance of three general-purpose scientific file formats—HDF5, netCDF4, and Zarr—across common operations.
• To identify which format offers the best performance for typical scientific data workflows involving dataset creation, writing, opening, and reading.
• To evaluate the impact of file format choice on performance in small-to-medium-sized datasets, representative of many scientific applications.
• To provide a reproducible benchmark framework for researchers to compare file formats under controlled conditions.
• To explore the influence of Python API overhead and caching on observed performance differences.

Proposed method

• A standardized benchmark suite was implemented in Python to measure time for create, open, write, read, and close operations on synthetic datasets.
• Datasets were generated with randomized data and stored in files using the native Python APIs for HDF5 (h5py), netCDF4 (netCDF4-python), and Zarr (zarr-python).
• The benchmark varied dataset count (1,024 to 8,192) and array dimensions (1D to 3D) to assess scalability and load effects.
• Each operation was timed over multiple runs and averaged to reduce noise from system-level variability.
• Results were visualized in bar graphs to compare relative performance across formats for each operation.
• The benchmark is open-source and hosted on GitHub for community use, extension, and contribution.

Experimental results

Research questions

• RQ1 Which file format—HDF5, netCDF4, or Zarr—performs fastest in creating a new dataset?
• RQ2 How do the three formats compare in terms of write performance when writing data to a dataset?
• RQ3 What is the relative performance of each format in opening a previously closed dataset?
• RQ4 Which format offers the fastest read performance when retrieving dataset values?
• RQ5 How does increasing the number of datasets affect the performance of each file format?

Key findings

• HDF5 demonstrated the fastest write performance, completing data writes in significantly less time than netCDF4 and Zarr.
• netCDF4 was the fastest at creating a new dataset, followed by HDF5 and then Zarr.
• For opening datasets, netCDF4 showed the lowest average time, followed by Zarr and then HDF5.
• HDF5 was the fastest at reading dataset values to standard output, with Zarr close behind and netCDF4 significantly slower.
• Increasing the number of datasets from 1,024 to 8,192 had minimal impact on average operation times, suggesting scalability is not a major bottleneck in this workload.
• Performance differences were likely influenced by Python API overhead and caching effects, which may diminish with larger data sizes in the gigabyte range.