[Paper Review] Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts

Dominik Scheinert, Lauritz Thamsen+5 more|arXiv (Cornell University)|Jul 29, 2021

Cloud Computing and Resource Management36 references17 citations

TL;DR

Bellamy proposes a black-box runtime prediction model for distributed dataflow jobs that reuses performance data across diverse execution contexts by incorporating scale-out, dataset size, and descriptive job properties. It uses a two-step approach—training a general model on multi-context data, then fine-tuning it for a specific context—achieving superior interpolation performance, especially for algorithms with non-trivial scale-out behavior, and enabling faster training via pre-trained weights.

ABSTRACT

Distributed dataflow systems enable the use of clusters for scalable data analytics. However, selecting appropriate cluster resources for a processing job is often not straightforward. Performance models trained on historical executions of a concrete job are helpful in such situations, yet they are usually bound to a specific job execution context (e.g. node type, software versions, job parameters) due to the few considered input parameters. Even in case of slight context changes, such supportive models need to be retrained and cannot benefit from historical execution data from related contexts. This paper presents Bellamy, a novel modeling approach that combines scale-outs, dataset sizes, and runtimes with additional descriptive properties of a dataflow job. It is thereby able to capture the context of a job execution. Moreover, Bellamy is realizing a two-step modeling approach. First, a general model is trained on all the available data for a specific scalable analytics algorithm, hereby incorporating data from different contexts. Subsequently, the general model is optimized for the specific situation at hand, based on the available data for the concrete context. We evaluate our approach on two publicly available datasets consisting of execution data from various dataflow jobs carried out in different environments, showing that Bellamy outperforms state-of-the-art methods.

Motivation & Objective

• . The paper aims to address the challenge of limited historical data for performance modeling in distributed dataflow systems.
• It seeks to improve runtime prediction accuracy by reusing data from related but different execution contexts.
• The objective is to reduce reliance on costly profiling or extensive historical data per context.
• It aims to support efficient resource configuration in cloud environments where users lack expertise or time for manual tuning.
• The goal is to enable scalable, accurate, and fast prediction for diverse users and dynamic cloud workloads.

Proposed method

• . Bellamy uses a two-step modeling approach: first training a general model on data from multiple contexts, then fine-tuning it for a specific context.
• It incorporates scale-out, dataset size, and additional descriptive job and resource properties as input features.
• The model is implemented as a neural network with task-specific components to handle context-aware prediction.
• Fine-tuning is performed using a combination of pre-trained weights and context-specific data, with variants including local, partial-unfreeze, and full-reset strategies.
• The approach leverages transfer learning principles to adapt a general model to a new context with minimal data.
• It uses early stopping and evaluation metrics like mean absolute error (MAE) to assess and optimize prediction performance.

Experimental results

Research questions

• RQ1 . Can a performance model trained on diverse execution contexts generalize effectively to a new, specific context with limited data?
• RQ2 . Does incorporating descriptive job and resource properties improve runtime prediction accuracy compared to models using only scale-out and dataset size?
• RQ3 . How does Bellamy’s cross-context learning capability compare to state-of-the-art methods in terms of prediction accuracy and training efficiency?
• RQ4 . What is the impact of different fine-tuning strategies (e.g., local, full-reset) on model performance and training time?
• RQ5 . Can Bellamy effectively predict runtimes for algorithms with non-trivial scale-out behavior, especially when data is scarce?

Key findings

• . Bellamy outperforms state-of-the-art methods in runtime prediction, particularly for algorithms with non-trivial scale-out behavior such as PageRank.
• . The local variant of Bellamy achieved the best average prediction performance across all evaluated algorithms (Grep, PageRank, SGD).
• . Models based on pre-trained weights reduced mean training time from 9.4 seconds (local variant) to between 2.8 and 3.8 seconds, significantly accelerating the process.
• . Pre-trained Bellamy models converged faster and terminated fine-tuning earlier, demonstrating improved training efficiency.
• . The approach enables effective interpolation across contexts, with significant performance gains observed when reusing models across different environments.
• . For algorithms with trivial scale-out behavior (e.g., linear), Bellamy did not consistently outperform simpler baselines, suggesting its advantage is most pronounced for complex scaling patterns.