[Paper Review] Bloscpack: a compressed lightweight serialization format for numerical data

Valentin Haenel|arXiv (Cornell University)|Apr 25, 2014

Algorithms and Data Compression3 references3 citations

TL;DR

Bloscpack is a fast, compressed, lightweight serialization format for numerical data that leverages the Blosc codec to enable high-speed compression and decompression of Numpy ndarrays. It outperforms NPZ and ZFile by up to 300× in compression speed on low-entropy datasets due to Blosc's multithreaded, cache-optimized, and shuffle-filtered compression, achieving both storage savings and performance gains in I/O-bound environments.

ABSTRACT

This paper introduces the Bloscpack file format and the accompanying Python reference implementation. Bloscpack is a lightweight, compressed binary file-format based on the Blosc codec and is designed for lightweight, fast serialization of numerical data. This article presents the features of the file-format and some some API aspects of the reference implementation, in particular the ability to handle Numpy ndarrays. Furthermore, in order to demonstrate its utility, the format is compared both feature- and performance-wise to a few alternative lightweight serialization solutions for Numpy ndarrays. The performance comparisons take the form of some comprehensive benchmarks over a range of different artificial datasets with varying size and complexity, the results of which are presented as the last section of this article.

Motivation & Objective

• To design a lightweight, compressed file format for efficient serialization of numerical data, particularly Numpy ndarrays.
• To reduce total serialization time by combining fast compression with I/O efficiency, leveraging Blosc's multithreaded and cache-optimized architecture.
• To outperform existing lightweight serialization formats (e.g., NPZ, ZFile, NPY) in both compression speed and I/O-bound scenarios.
• To evaluate the impact of Blosc’s shuffle filter and multithreading on compression performance across diverse data entropy levels and storage types.
• To provide a production-ready Python reference implementation that supports efficient, on-the-fly compression and decompression of numerical arrays.

Proposed method

• The Bloscpack format uses the Blosc codec as its core compression engine, which splits data into blocks for multithreaded processing.
• It applies a shuffle filter that reorders multi-byte elements (e.g., 64-bit doubles) by significance, improving compressibility for time-series and similar data.
• The format supports multiple compression codecs (blosclz, Snappy, LZ4, Zlib) through Blosc’s extensible meta-compressor interface.
• The reference implementation in Python natively handles Numpy ndarrays, enabling zero-copy serialization and efficient memory layout.
• Benchmarking is conducted across artificial datasets with varying sizes and entropy levels on SSD and SD storage to evaluate performance under different I/O conditions.
• Compression and decompression times are measured under both hot cache and cold I/O conditions to assess real-world performance.

Experimental results

Research questions

• RQ1 Can a compressed serialization format achieve faster total serialization time than uncompressed formats by reducing I/O and CPU overhead?
• RQ2 How does Bloscpack’s use of multithreaded, block-based compression with a shuffle filter affect performance on numerical datasets with varying entropy?
• RQ3 Does Bloscpack outperform NPZ and ZFile in compression speed, especially on low-entropy datasets where compression is most effective?
• RQ4 How does storage I/O speed (SSD vs. SD card) influence the performance advantage of Bloscpack over other formats?
• RQ5 Why do NPZ and ZFile exhibit unusually slow compression times on low-entropy datasets, and can Bloscpack’s approach mitigate this issue?

Key findings

• On low-entropy datasets, Bloscpack achieved a compression speed of 0.446 seconds, compared to 302 seconds for NPZ, representing a 300× performance improvement.
• Even on high-entropy datasets with minimal compressibility, Bloscpack remained significantly faster than NPZ and ZFile due to its ability to quickly detect and copy uncompressed data.
• On SD storage (I/O-bound), Bloscpack was up to an order of magnitude faster than NPY for medium and large low-entropy datasets, due to reduced I/O from higher compression.
• Bloscpack at level 7 was twice as fast as NPY during compression on medium-entropy, medium-sized datasets when using SD storage.
• NPZ and ZFile exhibited abnormally slow compression on low-entropy data, likely due to the Lempel-Ziv complexity of such data under DEFLATE, which lacks a shuffle filter.
• Bloscpack achieved better compression ratios than NPY on low-entropy data, and better than NPZ and ZFile on medium-entropy data, with ratios approaching zero on high-entropy data.