[Paper Review] Data Compression for Analytics over Large-scale In-memory Column Databases

Chunbin Lin, Jianguo Wang+1 more|arXiv (Cornell University)|Jun 30, 2016

Advanced Data Storage Technologies8 references3 citations

TL;DR

This paper investigates the viability of data compression in large-scale in-memory columnar databases for OLAP workloads. It argues that certain compression techniques—like dictionary encoding, run-length encoding (RLE), and bitmap encoding—can accelerate query processing by enabling direct operation on compressed data, reducing both memory usage and I/O. The key contribution is a decision framework for selecting optimal compression schemes based on data characteristics, with vsb-RLE and compressed bitmaps showing superior performance and space efficiency.

ABSTRACT

Data compression schemes have exhibited their importance in column databases by contributing to the high-performance OLAP (Online Analytical Processing) query processing. Existing works mainly concentrate on evaluating compression schemes for disk-resident databases as data is mostly stored on disks. With the continuously decreasing of the price/capacity ratio of main memory, it is the tendencies of the times to reside data in main memory. But the discussion of data compression on in-memory databases is very vague in the literature. In this work, we present an updated discussion about whether it is valuable to use data compression techniques in memory databases. If yes, how should memory databases apply data compression schemes to maximize performance?

Motivation & Objective

• To determine whether data compression remains beneficial in in-memory column databases despite reduced I/O costs.
• To identify which compression schemes enable direct query processing on compressed data without decompression, thus improving performance.
• To provide a systematic framework for selecting optimal compression techniques based on column type (integer/string), sorting, and domain size.
• To evaluate and optimize compression schemes such as vsb-RLE and compressed bitmaps for in-memory OLAP workloads.
• To demonstrate that some compression methods can simultaneously reduce memory footprint and accelerate query execution in main memory databases.

Proposed method

• Analyzes widely used compression schemes—dictionary encoding, run-length encoding (RLE), bitmap encoding, and Huffman encoding—within the context of in-memory column stores.
• Evaluates each scheme's ability to support direct query processing on compressed data, distinguishing between schemes that allow in-situ operations and those requiring full decompression.
• Proposes vsb-RLE (value-size-bit dictionary RLE), a hybrid compression method combining variable-size RLE with bit-aligned dictionary encoding for integer columns, especially effective on sorted data.
• Introduces bit-DICT optimization to further compress RLE-encoded values by encoding only the values (not run lengths), preserving performance while reducing space.
• Employs experimental evaluation to measure decompression time and space savings, particularly for Huffman encoding under Zipf-distributed data.
• Develops a decision tree (Figure 2) to guide selection of compression schemes based on column type, sorting status, and domain size.

Experimental results

Research questions

• RQ1 Is data compression still beneficial in in-memory column databases where I/O is no longer the primary bottleneck?
• RQ2 Which compression schemes allow direct query processing on compressed data without decompression, and thus can improve performance?
• RQ3 How can compression schemes be optimized for integer-type columns in in-memory databases to reduce space while maintaining fast query execution?
• RQ4 What are the performance and space trade-offs of using Huffman encoding in in-memory column stores?
• RQ5 What criteria should guide the selection of an optimal compression scheme for a given column in an in-memory OLAP database?

Key findings

• Data compression remains beneficial in in-memory column databases not only for reducing memory footprint but also for improving query performance when using compression schemes that support direct operation on compressed data.
• Dictionary encoding, RLE (including vsb-RLE), and compressed bitmap encoding allow queries to be processed directly on compressed data, reducing data movement and accelerating execution compared to uncompressed data.
• vsb-RLE achieves a space cost of 6.5MB for a 1-billion-row integer column with 1 million distinct values—reducing storage by 18% compared to standard vs-RLE (8MB) and 15% compared to vsl-RLE (12MB).
• Huffman encoding is not recommended for in-memory databases due to high decompression overhead: it took 3.2 minutes to decompress a 1-million-row column under Zipf distribution.
• Compressed bitmap encoding is effective only for columns with small domain sizes (e.g., <50), as the number of bit-vectors remains large and memory-intensive for high-cardinality columns.
• The proposed decision tree (Figure 2) provides a practical, data-driven guide for selecting compression schemes: use vsb-RLE for sorted integer columns, compressed bitmaps for small domains, and bit-DICT for large domains.