[Paper Review] In-Memory Database Systems - A Paradigm Shift

Mohit Kumar Gupta, Vishal Verma+1 more|arXiv (Cornell University)|Feb 6, 2014

Cloud Computing and Resource Management7 citations

TL;DR

This paper proposes in-memory database systems (IMDS) as a paradigm shift to address the limitations of traditional on-disk databases in handling real-time, high-volume data workloads. By storing data primarily in main memory, IMDS enables sub-second response times and high-throughput processing, overcoming I/O bottlenecks; the study analyzes design challenges and evaluates leading commercial and open-source IMDS solutions like SAP HANA and VoltDB.

ABSTRACT

In today world, organizations like Google, Yahoo, Amazon, Facebook etc. are facing drastic increase in data. This leads to the problem of capturing, storing, managing and analyzing terabytes or petabytes of data, stored in multiple formats, from different internal and external sources. Moreover, new applications scenarios like weather forecasting, trading, artificial intelligence etc. need huge data processing in real time. These requirements exceed the processing capacity of traditional on-disk database management systems to manage this data and to give speedy real time results. Therefore, data management needs new solutions for coping with the challenges of data volumes and processing data in real-time. An in-memory database system (IMDS) is a latest breed of database management system which is becoming answer to above challenges with other supporting technologies. IMDS is capable to process massive data distinctly faster. This paper explores IMDS approach and its associated design issues and challenges. It also investigates some famous commercial and open-source IMDS solutions available in the market.

Motivation & Objective

• Address the growing challenge of managing and processing terabyte- and petabyte-scale data in real time.
• Identify the limitations of traditional on-disk database management systems in handling modern data-intensive workloads.
• Explain how in-memory database systems (IMDS) overcome I/O bottlenecks through main memory storage.
• Investigate key design issues and challenges in implementing scalable and efficient in-memory database systems.
• Evaluate prominent commercial and open-source IMDS solutions available in the market as practical implementations of the paradigm shift.

Proposed method

• Analyzes the architectural shift from on-disk to in-memory storage as a core enabler for low-latency data processing.
• Examines the role of main memory in eliminating disk I/O bottlenecks, enabling faster data access and processing.
• Reviews design considerations such as data compression, logging, durability, and concurrency control in in-memory systems.
• Compares commercial and open-source IMDS platforms based on performance, scalability, and feature sets.
• Uses system-level analysis to evaluate how in-memory processing supports real-time applications like AI, trading, and weather forecasting.
• Cites industry examples (e.g., SAP HANA, VoltDB) to illustrate practical deployment and performance gains.

Experimental results

Research questions

• RQ1 How do in-memory database systems overcome the performance limitations of traditional on-disk databases?
• RQ2 What are the key architectural and design challenges in building scalable and durable in-memory database systems?
• RQ3 What are the performance and scalability characteristics of leading commercial and open-source in-memory database solutions?
• RQ4 In what real-time application domains do in-memory databases provide a significant advantage over traditional systems?
• RQ5 How does in-memory storage impact data processing speed, especially for analytical and transactional workloads?

Key findings

• In-memory database systems achieve significantly faster data processing by eliminating disk I/O latency, enabling real-time analytics and transaction processing.
• IMDS platforms such as SAP HANA and VoltDB demonstrate sub-second response times for complex queries on large-scale data.
• The use of main memory as primary storage allows for higher throughput and lower latency compared to traditional on-disk systems.
• Design challenges such as data durability, logging, and recovery mechanisms remain critical and require specialized solutions in in-memory systems.
• The paper confirms that in-memory databases are a viable and necessary paradigm shift for modern data-intensive applications requiring real-time processing.
• Commercial and open-source IMDS solutions show strong performance in high-volume, low-latency environments such as financial trading and AI workloads.