[Paper Review] Exploiting Reuse in Pipeline-Aware Hyperparameter Tuning

Liam Li, Evan Sparks+2 more|arXiv (Cornell University)|Mar 12, 2019

Machine Learning and Data Classification28 references4 citations

TL;DR

This paper proposes a pipeline-aware hyperparameter tuning framework that exploits computational reuse across multi-stage machine learning pipelines by optimizing design, balancing computation, and improving caching. It introduces gridded random search for shared prefix discovery, early-stopping via Successive Halving to reduce training time, and a novel ILP-based caching strategy; experiments show over 17× speedup on speech classification and 70× on text classification by reusing shared pipeline stages.

ABSTRACT

Hyperparameter tuning of multi-stage pipelines introduces a significant computational burden. Motivated by the observation that work can be reused across pipelines if the intermediate computations are the same, we propose a pipeline-aware approach to hyperparameter tuning. Our approach optimizes both the design and execution of pipelines to maximize reuse. We design pipelines amenable for reuse by (i) introducing a novel hybrid hyperparameter tuning method called gridded random search, and (ii) reducing the average training time in pipelines by adapting early-stopping hyperparameter tuning approaches. We then realize the potential for reuse during execution by introducing a novel caching problem for ML workloads which we pose as a mixed integer linear program (ILP), and subsequently evaluating various caching heuristics relative to the optimal solution of the ILP. We conduct experiments on simulated and real-world machine learning pipelines to show that a pipeline-aware approach to hyperparameter tuning can offer over an order-of-magnitude speedup over independently evaluating pipeline configurations.

Motivation & Objective

• To address the high computational cost of tuning multi-stage machine learning pipelines by exploiting reuse across configurations.
• To reduce redundant computation in hyperparameter tuning by merging pipelines with shared intermediate stages into a single DAG.
• To improve efficiency by balancing expensive training stages through early-stopping and partial training strategies.
• To develop a cache-aware strategy tailored for ML workloads with variable node costs, going beyond standard heuristics like LRU.
• To evaluate and compare advanced caching heuristics against an optimal ILP solution for ML pipeline execution.

Proposed method

• Introduces gridded random search—a hybrid method that limits branching factors of continuous hyperparameters to promote shared prefixes in pipeline DAGs.
• Applies Successive Halving as an early-stopping strategy to break down expensive training nodes into cheaper, sequential steps, improving DAG balance.
• Models the caching problem for ML pipelines as a cache-dependent generalized paging problem, formulated as a mixed-integer linear program (ILP).
• Proposes and evaluates new caching heuristics—RECIPROCAL and WRECIPROCAL—designed to avoid caching high-cost, low-value nodes.
• Uses the optimal ILP solution as a benchmark to evaluate heuristic performance, particularly in memory-constrained settings.
• Employs a three-pronged approach: design for reuse, balance computation, and intelligent caching to maximize efficiency.

Experimental results

Research questions

• RQ1 Can shared prefixes in hyperparameter configurations be systematically exploited to reduce redundant computation in pipeline tuning?
• RQ2 How can early-stopping and partial training strategies improve the balance of computation in ML pipelines to enhance reuse benefits?
• RQ3 Is the standard LRU cache eviction policy suitable for machine learning pipelines with variable node costs?
• RQ4 Can a novel ILP-based caching strategy outperform existing heuristics in reducing total execution time for pipeline hyperparameter tuning?
• RQ5 To what extent can a pipeline-aware approach achieve speedups in real-world ML workloads with diverse computational profiles?

Key findings

• On the 20 Newsgroups dataset, the proposed pipeline-aware approach achieved a 70× speedup over independent configuration evaluation using caching heuristics.
• For the Amazon Reviews dataset, all caching strategies delivered over 40× speedup, demonstrating strong reuse benefits in front-loaded pipelines.
• LRU performed poorly in small-memory settings due to eviction of expensive early-stage nodes, highlighting its unsuitability for ML pipelines.
• WRECIPROCAL outperformed RECIPROCAL and LRU by being less likely to cache large, low-value nodes, making it more robust across cache sizes.
• On the TIMIT speech classification task, using gridded random search and Successive Halving led to over 17× speedup when combined with caching under realistic settings.
• The optimal ILP solution served as a strong benchmark, showing that heuristic-based caching can achieve near-optimal performance when properly designed.