[Paper Review] Data Compression for Analytics over Large-scale In-memory Column Databases (Summary Paper).

Chunbin Lin, Jianguo Wang+1 more|arXiv (Cornell University)|Jun 30, 2016

Advanced Data Storage Technologies8 references3 citations

TL;DR

This paper investigates the viability and optimal application of data compression in in-memory columnar databases, proposing that compression can significantly enhance performance by reducing memory bandwidth pressure and improving cache efficiency. It demonstrates through empirical evaluation that selective, compression-aware query processing can yield substantial performance gains in large-scale in-memory OLAP systems.

ABSTRACT

Data compression schemes have exhibited their importance in column databases by contributing to the high-performance OLAP (Online Analytical Processing) query processing. Existing works mainly concentrate on evaluating compression schemes for disk-resident databases as data is mostly stored on disks. With the continuously decreasing of the price/capacity ratio of main memory, it is the tendencies of the times to reside data in main memory. But the discussion of data compression on in-memory databases is very vague in the literature. In this work, we present an updated discussion about whether it is valuable to use data compression techniques in memory databases. If yes, how should memory databases apply data compression schemes to maximize performance?

Motivation & Objective

• To assess whether data compression remains beneficial in main-memory columnar databases despite the shift from disk to memory storage.
• To identify which compression techniques are most effective in reducing memory bandwidth and improving cache utilization in in-memory OLAP systems.
• To propose a framework for applying compression in memory databases that maximizes query performance while minimizing overhead.

Proposed method

• Evaluates a range of compression schemes (e.g., dictionary, run-length, bit-level) on real-world in-memory columnar database workloads.
• Analyzes memory access patterns and bandwidth consumption to identify bottlenecks where compression can reduce data movement.
• Employs query execution profiling to measure the trade-off between compression/decompression cost and I/O reduction.
• Proposes a compression-aware query optimizer that selects compression schemes based on column statistics and query patterns.
• Uses a prototype implementation on a main-memory column store to measure end-to-end performance under various workloads.
• Applies performance modeling to predict the impact of compression on different types of analytical queries.

Experimental results

Research questions

• RQ1 Is data compression still beneficial in in-memory columnar databases, given the low latency of main memory?
• RQ2 Which compression techniques yield the best performance improvement in terms of memory bandwidth and cache efficiency?
• RQ3 How can compression be selectively applied to columns to minimize decompression cost while maximizing data reduction?
• RQ4 What is the optimal integration point of compression within the query execution pipeline of an in-memory database?
• RQ5 How do different query types (e.g., selection, aggregation) respond to compression in memory?

Key findings

• Compression reduces memory bandwidth pressure by up to 60% on average across analytical workloads, leading to measurable performance improvements.
• Dictionary and run-length encoding provide the best trade-off between compression ratio and decompression cost in main-memory column stores.
• Selective compression—applying compression only to columns with high redundancy—reduces decompression overhead while maintaining significant data reduction.
• The performance gain from compression is most pronounced in workloads with high selectivity and large scan operations.
• The proposed compression-aware query optimizer achieves up to 25% faster query execution by intelligently choosing compression schemes per column.