[Paper Review] Measuring the Optimality of Hadoop Optimization

Woocheol Kim, Changryong Baek+1 more|arXiv (Cornell University)|Jul 10, 2013

Cloud Computing and Resource Management16 references3 citations

TL;DR

This paper proposes a novel measure, $vet_{job}$, to quantify how close a Hadoop job is to its theoretical performance lower bound by analyzing hardware utilization and processing time overheads. By estimating the ideal running time through statistical profiling of record processing, the method enables precise assessment of optimization effectiveness, with $vet_{job} = 1$ indicating perfect optimization, and demonstrates high accuracy across diverse cluster configurations.

ABSTRACT

In recent years, much research has focused on how to optimize Hadoop jobs. Their approaches are diverse, ranging from improving HDFS and Hadoop job scheduler to optimizing parameters in Hadoop configurations. Despite their success in improving the performance of Hadoop jobs, however, very little is known about the limit of their optimization performance. That is, how optimal is a given Hadoop optimization? When a Hadoop optimization method X improves the performance of a job by Y %, how do we know if this improvement is as good as it can be? To answer this question, in this paper, we first examine the ideal best case, the lower bound, of running time for Hadoop jobs and develop a measure to accurately estimate how optimal a given Hadoop optimization is with respect to the lower bound. Then, we demonstrate how one may exploit the proposed measure to improve the optimization of Hadoop jobs.

Motivation & Objective

• To address the lack of a standardized metric for evaluating how close a Hadoop optimization is to its theoretical performance limit.
• To identify and estimate the lower bound of Hadoop job running time, accounting for hardware utilization and processing overheads.
• To develop a practical, accurate measure that reflects the degree of optimization relative to the ideal case, independent of specific optimizers.
• To complement existing tools like Starfish by providing a benchmark for optimization potential.
• To guide Hadoop practitioners in identifying untapped performance improvement opportunities.

Proposed method

• Profile detailed processing times of individual data records in Hadoop jobs to balance accuracy and profiling overhead.
• Apply statistical methods to identify and exclude outliers in processing times, focusing on normal-case performance.
• Estimate the ideal processing time per record by extrapolating the normal-case processing time distribution.
• Separately estimate the ideal running time for map and reduce tasks based on hardware and system-level overheads.
• Combine map and reduce task estimates into a unified measure, $vet_{job}$, representing the job's optimization degree relative to the theoretical lower bound.
• Validate the measure using real experiments across diverse cluster setups and Hadoop configurations.

Experimental results

Research questions

• RQ1 What is the theoretical lower bound of running time for a Hadoop job, given hardware and system constraints?
• RQ2 How can the ideal performance of a Hadoop job be estimated with minimal profiling overhead while maintaining high accuracy?
• RQ3 To what extent can existing optimization techniques be evaluated in terms of their proximity to the theoretical performance limit?
• RQ4 How does the proposed $vet_{job}$ measure compare to existing optimization evaluation methods in terms of accuracy and practical utility?
• RQ5 Can the proposed measure effectively guide further optimization beyond current tools like Starfish?

Key findings

• The proposed $vet_{job}$ measure accurately estimates the degree of optimization of a Hadoop job, with $vet_{job} = 1$ indicating perfect optimization relative to the theoretical lower bound.
• Even after aggressive optimization by tools like Starfish, real Hadoop jobs often show significant room for improvement, as evidenced by $vet_{job}$ values substantially below 1.
• The method successfully identifies that CPU utilization and context-switching overheads are critical but often overlooked factors in performance estimation.
• The statistical profiling approach effectively filters out abnormal processing times, enabling reliable estimation of normal-case processing time.
• Experiments across diverse cluster settings confirm that $vet_{job}$ provides a consistent and accurate benchmark for optimization effectiveness.
• The measure complements existing tools such as Starfish by offering an independent, objective standard for evaluating optimization gains.