[Paper Review] Algorithm Runtime Prediction: Methods & Evaluation

Frank Hutter, Lin Xu+2 more|arXiv (Cornell University)|Nov 5, 2012

Advanced Multi-Objective Optimization Algorithms88 references44 citations

TL;DR

This paper presents a comprehensive framework for predicting algorithm runtime using machine learning, introducing advanced models based on random forests and approximate Gaussian processes, along with novel instance features for SAT, TSP, and MIP problems. It demonstrates that these models significantly outperform prior methods in generalizing to new instances and parameter configurations across 11 algorithms and 35 instance distributions.

ABSTRACT

Perhaps surprisingly, it is possible to predict how long an algorithm will take to run on a previously unseen input, using machine learning techniques to build a model of the algorithm's runtime as a function of problem-specific instance features. Such models have important applications to algorithm analysis, portfolio-based algorithm selection, and the automatic configuration of parameterized algorithms. Over the past decade, a wide variety of techniques have been studied for building such models. Here, we describe extensions and improvements of existing models, new families of models, and -- perhaps most importantly -- a much more thorough treatment of algorithm parameters as model inputs. We also comprehensively describe new and existing features for predicting algorithm runtime for propositional satisfiability (SAT), travelling salesperson (TSP) and mixed integer programming (MIP) problems. We evaluate these innovations through the largest empirical analysis of its kind, comparing to a wide range of runtime modelling techniques from the literature. Our experiments consider 11 algorithms and 35 instance distributions; they also span a very wide range of SAT, MIP, and TSP instances, with the least structured having been generated uniformly at random and the most structured having emerged from real industrial applications. Overall, we demonstrate that our new models yield substantially better runtime predictions than previous approaches in terms of their generalization to new problem instances, to new algorithms from a parameterized space, and to both simultaneously.

Motivation & Objective

• To improve the accuracy and generalization of algorithm runtime prediction models for highly parameterized algorithms.
• To develop new, comprehensive feature sets for SAT, MIP, and TSP problems, including probing and timing features.
• To conduct the most extensive empirical evaluation of runtime prediction methods to date, covering novel instances, novel parameter configurations, and both simultaneously.
• To integrate survival analysis techniques to better handle censored runtime data from prematurely terminated runs.
• To establish a new state-of-the-art in empirical performance modeling (EPMs) for algorithm selection, configuration, and benchmark generation.

Proposed method

• Employ random forests and approximate Gaussian processes as core regression models for runtime prediction.
• Integrate algorithm parameters (both categorical and continuous) as input features to model performance across parameterized algorithm spaces.
• Design and implement novel instance features: probing features (e.g., partial solving behavior) and timing features (e.g., early-stage execution metrics).
• Use a unified framework to evaluate models across three distinct prediction tasks: generalization to new instances, new parameter settings, and both.
• Apply survival analysis techniques (e.g., Cox proportional hazards models) to handle censored data from runs terminated before completion.
• Leverage the full dataset of 11 algorithms and 35 instance distributions across SAT, MIP, and TSP to train and validate models.

Experimental results

Research questions

• RQ1 Can advanced machine learning models like random forests and approximate Gaussian processes improve runtime prediction accuracy compared to classical regression models?
• RQ2 To what extent do novel instance features—especially probing and timing features—enhance predictive performance across SAT, MIP, and TSP problems?
• RQ3 How well do the proposed models generalize to unseen problem instances, unseen algorithm parameter configurations, and both simultaneously?
• RQ4 Can survival analysis techniques improve prediction performance when dealing with censored runtime data from early-terminated runs?
• RQ5 What is the current state of the art in empirical performance modeling (EPMs), and how do the proposed models compare to all existing methods in a comprehensive evaluation?

Key findings

• The proposed models, particularly random forests with survival analysis extensions, achieve substantially better generalization performance than previous approaches on unseen problem instances.
• The integration of algorithm parameters as model inputs significantly improves prediction accuracy, especially in the context of parameterized algorithm configuration.
• The new feature sets—138 for SAT, 121 for MIP, and 64 for TSP—demonstrate that probing and timing features are highly predictive of runtime behavior.
• The evaluation on 11 algorithms and 35 instance distributions confirms that the proposed models outperform all prior methods in terms of predictive accuracy and robustness.
• Survival analysis techniques further improve the performance of the best model (random forests), especially in handling censored data from incomplete runs.
• The study establishes a new benchmark for EPMs, showing that the combination of advanced models, rich features, and comprehensive evaluation leads to state-of-the-art performance in algorithm runtime prediction.