[Paper Review] PerfXplain: Debugging MapReduce Job Performance

Nodira Khoussainova, Magdalena Bałazińska+1 more|arXiv (Cornell University)|Mar 29, 2012

Data Quality and Management26 references4 citations

TL;DR

PerfXplain is a system that helps non-expert users debug MapReduce job performance by generating human-readable explanations for performance differences between job pairs. It uses a query language (PXQL) and machine learning on execution logs to produce explanations with high relevance, precision, and generality, outperforming naive baselines in experiments on AWS EC2 logs.

ABSTRACT

While users today have access to many tools that assist in performing large scale data analysis tasks, understanding the performance characteristics of their parallel computations, such as MapReduce jobs, remains difficult. We present PerfXplain, a system that enables users to ask questions about the relative performances (i.e., runtimes) of pairs of MapReduce jobs. PerfXplain provides a new query language for articulating performance queries and an algorithm for generating explanations from a log of past MapReduce job executions. We formally define the notion of an explanation together with three metrics, relevance, precision, and generality, that measure explanation quality. We present the explanation-generation algorithm based on techniques related to decision-tree building. We evaluate the approach on a log of past executions on Amazon EC2, and show that our approach can generate quality explanations, outperforming two naive explanation-generation methods.

Motivation & Objective

• To address the challenge of debugging performance issues in MapReduce jobs for non-expert users.
• To enable users to ask comparative performance questions about pairs of jobs or tasks.
• To generate high-quality, human-interpretable explanations for performance anomalies.
• To formalize explanation quality using relevance, precision, and generality metrics.
• To develop a scalable, log-based explanation generation algorithm that requires minimal user input.

Proposed method

• Uses a custom query language, PerfXplain Query Language (PXQL), to specify comparative performance questions.
• Represents each job or task execution as a feature vector of configuration and performance metrics.
• Applies machine learning techniques similar to decision tree induction to mine explanations from historical execution logs.
• Generates explanations as two predicates: a 'despite' clause (expected behavior context) and a 'because' clause (observed deviation explanation).
• Optimizes explanation quality using three metrics: relevance (contextual fit), precision (accuracy of explanation), and generality (applicability across similar cases).
• Leverages past execution logs to identify positive (expected behavior) and negative (unexpected behavior) example pairs for training.

Experimental results

Research questions

• RQ1 How can users effectively ask and answer comparative performance questions about MapReduce jobs without expert systems knowledge?
• RQ2 What constitutes a high-quality explanation for performance differences between job executions?
• RQ3 How can explanation quality be formally measured and optimized using relevance, precision, and generality?
• RQ4 Can machine learning techniques from log data generate explanations that outperform naive heuristics?
• RQ5 To what extent is the approach generalizable beyond MapReduce to other parallel data processing systems?

Key findings

• PerfXplain successfully generates explanations that are more relevant, precise, and general than two naive baselines in evaluation on real AWS EC2 logs.
• The system's explanation generation algorithm effectively identifies contextual factors (despite clauses) and root causes (because clauses) for performance deviations.
• The use of feature vectors enables the approach to generalize beyond MapReduce to other parallel processing systems.
• The formal metrics of relevance, precision, and generality provide a principled way to evaluate and improve explanation quality.
• Experiments show that PerfXplain's explanations are not only accurate but also meaningful and actionable for non-expert users.