[Paper Review] Large-scale Data Modelling in Hive and Distributed Query Processing using MapReduce and Tez

Abzetdin Adamov|arXiv (Cornell University)|Jan 29, 2023

Cloud Computing and Resource Management4 citations

TL;DR

This paper investigates Apache Hive's role in large-scale data modeling and distributed query processing using MapReduce and Tez engines within the Hadoop ecosystem. It evaluates Hive's data definition and manipulation capabilities, demonstrating that Tez significantly improves query processing performance over MapReduce due to its optimized execution engine and reduced I/O overhead.

ABSTRACT

Huge amounts of data being generated continuously by digitally interconnected systems of humans, organizations and machines. Data comes in variety of formats including structured, unstructured and semi-structured, what makes it impossible to apply the same standard approaches, techniques and algorithms to manage and process this data. Fortunately, the enterprise level distributed platform named Hadoop Ecosystem exists. This paper explores Apache Hive component that provides full stack data managements functionality in terms of Data Definition, Data Manipulation and Data Processing. Hive is a data warehouse system, which works with structured data stored in tables. Since, Hive works on top the Hadoop HDSFS, it benefits from extraordinary feature of HDFS including Fault Tolerance, Reliability, High Availability, Scalability, etc. In addition, Hive can take advantage of distributed computing power of the cluster through assigning jobs to MapReduce, Tez and Spark engines to run complex queries. The paper is focused on studying of Hive Data Model and analysis of processing performance done by MapReduce and Tez.

Motivation & Objective

• To analyze Hive's data modeling capabilities for structured, semi-structured, and unstructured data in large-scale environments.
• To evaluate the performance differences between MapReduce and Tez as execution engines for Hive queries.
• To assess how Hive leverages HDFS for fault tolerance, scalability, and high availability in distributed data processing.
• To identify architectural advantages of Tez over MapReduce in complex query execution workflows.

Proposed method

• Utilized Apache Hive as a data warehouse system to manage structured data through SQL-like DDL and DML operations.
• Executed complex analytical queries using both MapReduce and Tez engines on a Hadoop cluster with HDFS storage.
• Measured query execution time, I/O operations, and job completion rates to compare performance across engines.
• Analyzed job execution plans and DAG structures to understand optimization differences between MapReduce and Tez.
• Used real-world data workloads to benchmark query performance under varying data sizes and complexity.
• Evaluated the impact of fault tolerance, scalability, and high availability features provided by HDFS and Hive integration.

Experimental results

Research questions

• RQ1 How does Hive support large-scale data modeling across structured, semi-structured, and unstructured data formats?
• RQ2 What are the performance differences between MapReduce and Tez when processing complex Hive queries?
• RQ3 How does Hive leverage HDFS features such as fault tolerance and scalability in distributed query execution?
• RQ4 To what extent does Tez reduce I/O overhead and improve job execution efficiency compared to MapReduce?
• RQ5 What architectural advantages does Tez provide in optimizing complex query execution workflows in Hive?

Key findings

• Tez outperformed MapReduce in query execution time, particularly for complex, multi-stage queries, due to its efficient DAG-based execution model.
• MapReduce exhibited higher I/O overhead due to intermediate data spillage to disk between map and reduce phases.
• Hive effectively abstracted complex data processing logic, enabling SQL-like operations on large-scale data stored in HDFS.
• The integration of Hive with HDFS ensured high availability, fault tolerance, and horizontal scalability across cluster nodes.
• Tez reduced the number of intermediate job stages, leading to faster job completion and lower resource consumption.
• Performance benchmarks showed a measurable improvement in throughput and latency when using Tez over MapReduce for ETL and analytical workloads.