[Paper Review] Multitask and Transfer Learning for Autotuning Exascale Applications

Wissam M. Sid-Lakhdar, Mohsen Mahmoudi Aznaveh+2 more|arXiv (Cornell University)|Aug 15, 2019

Machine Learning and Data Classification29 references4 citations

TL;DR

This paper proposes a multitask and transfer learning framework for autotuning exascale applications, leveraging knowledge from prior tuning tasks to accelerate optimization. It achieves a 1.5x average runtime improvement over state-of-the-art autotuners like OpenTuner and HpBandSter, with low-budget configurations outperforming standard methods even under stringent resource constraints.

ABSTRACT

Multitask learning and transfer learning have proven to be useful in the field of machine learning when additional knowledge is available to help a prediction task. We aim at deriving methods following these paradigms for use in autotuning, where the goal is to find the optimal performance parameters of an application treated as a black-box function. We show comparative results with state-of-the-art autotuning techniques. For instance, we observe an average $1.5x$ improvement of the application runtime compared to the OpenTuner and HpBandSter autotuners. We explain how our approaches can be more suitable than some state-of-the-art autotuners for the tuning of any application in general and of expensive exascale applications in particular.

Motivation & Objective

• Address the challenge of optimizing expensive exascale applications with limited tuning budget due to high computational cost per run.
• Overcome the inefficiency of brute-force grid search and standard autotuners in high-cost exascale environments.
• Enable faster, more accurate autotuning by reusing knowledge from previously tuned problems across diverse workloads.
• Develop a scalable, adaptive framework that supports multiple tuning metrics, resource constraints, and incremental learning.
• Improve performance portability and long-term tuning efficiency across evolving hardware and application workloads.

Proposed method

• Introduce a multitask learning approach (MLA) that jointly optimizes multiple problem instances simultaneously, sharing information across tasks to improve convergence.
• Develop transfer learning algorithms (TLA1 and TLA2) that leverage pre-trained models from prior tuning tasks to accelerate new tuning tasks with minimal or zero new runs.
• Use Gaussian process-based surrogate models to represent the performance landscape of tuning parameters, enabling efficient Bayesian optimization.
• Design a black-box autotuning interface exposing input parameters, resource constraints, and optimization metrics to enable modular, composable tuning across application components.
• Implement a hierarchical tuning pipeline where initial model training (MLA) is followed by fast, low-cost inference (TLA1/TLA2) for new tasks.
• Incorporate transfer mechanisms that adapt pre-trained models to new problem instances using few or no additional evaluations, reducing reliance on expensive function calls.

Experimental results

Research questions

• RQ1 Can multitask learning improve the efficiency and accuracy of autotuning across diverse problem instances in exascale applications?
• RQ2 To what extent can transfer learning reduce the number of expensive application runs required for optimal parameter configuration?
• RQ3 How do multitask and transfer learning methods compare to state-of-the-art autotuners like OpenTuner and HpBandSter under low-budget conditions?
• RQ4 Can knowledge from prior tuning tasks be effectively transferred to new, unseen problem instances without retraining from scratch?
• RQ5 How do these methods perform in the presence of noisy, non-stationary, and discontinuous performance metrics typical of exascale workloads?

Key findings

• The proposed multitask learning approach (MLA) outperforms OpenTuner in 66% of tested cases, with up to a 40% improvement in application runtime.
• TLA1, a run-free transfer learning method, achieves competitive performance with configurations that match or exceed those found by OpenTuner and HpBandSter.
• TLA2, using only 100 application runs, achieves a best runtime of 5.67e-3 seconds on a 500×500 ScaLAPACK QR factorization, outperforming OpenTuner with 100 runs (7.00e-3 s) and approaching grid search (5.15e-3 s).
• The framework reduces average application runtime by 1.5x compared to OpenTuner and HpBandSter, demonstrating significant efficiency gains.
• The methods are particularly effective under low-budget conditions, making them more suitable than standard autotuners for expensive exascale applications.
• Experimental results confirm that transfer learning enables accurate tuning predictions with minimal or no new evaluations, validating the framework’s scalability and adaptability.