[Paper Review] Big Data Technology Literature Review

Michael Bar-Sinai|arXiv (Cornell University)|Jun 30, 2015

Cloud Computing and Resource Management4 citations

TL;DR

This literature review provides a comprehensive survey of big data technologies, focusing on distributed storage and processing systems such as Hadoop, NoSQL databases, and stream processing frameworks. It outlines key algorithms and architectures, links to open-source implementations, and serves as a foundational reference for researchers exploring scalable data management solutions.

ABSTRACT

A short overview of various algorithms and technologies that are helpful for big data storage and manipulation. Includes pointers to papers for further reading, and, where applicable, pointers to open source projects implementing a described storage type.

Motivation & Objective

• To provide a structured overview of emerging big data technologies for distributed storage and processing.
• To identify and categorize key algorithms and system architectures relevant to big data workloads.
• To guide researchers toward foundational papers and open-source implementations for further study.
• To bridge the gap between theoretical big data concepts and practical deployment through technology mapping.
• To serve as a reference for researchers seeking entry points into the evolving big data ecosystem.

Proposed method

• Systematic review of academic literature and technical designs in distributed computing and database systems.
• Categorization of big data technologies into storage, processing, and stream processing paradigms.
• Mapping of each technology to its underlying algorithms and architectural principles.
• Inclusion of references to seminal papers and open-source projects for each technology reviewed.
• Use of visual diagrams (11 figures) to illustrate system components and data flow in big data pipelines.
• Focus on production-ready systems such as Hadoop, HBase, Cassandra, and Spark, with emphasis on scalability and fault tolerance.

Experimental results

Research questions

• RQ1 What are the core architectural patterns underlying modern big data storage and processing systems?
• RQ2 How do NoSQL databases differ in design and performance from traditional relational databases in big data workloads?
• RQ3 What are the key characteristics of stream processing frameworks that enable real-time data analysis?
• RQ4 Which open-source implementations best represent the state-of-the-art in big data technology?
• RQ5 How do distributed computing models like MapReduce and Spark improve scalability over monolithic systems?

Key findings

• Hadoop MapReduce remains a foundational model for batch processing, enabling fault-tolerant, scalable data processing across clusters.
• NoSQL databases such as Cassandra and HBase provide high availability and partition tolerance, making them suitable for large-scale, write-intensive workloads.
• Stream processing frameworks like Storm and Spark Streaming enable low-latency, real-time data processing with high throughput.
• The integration of in-memory computation in systems like Apache Spark significantly improves performance over disk-based MapReduce.
• Open-source implementations such as Hadoop, Cassandra, and Spark are widely adopted and actively maintained, forming the backbone of modern big data platforms.
• The literature review identifies a clear trend toward hybrid architectures combining batch and stream processing for full-stack data pipeline support.