[Paper Review] ASlib: A Benchmark Library for Algorithm Selection

Bernd Bischl, Pascal Kerschke+9 more|arXiv (Cornell University)|Jun 8, 2015

Machine Learning and Data Classification72 references4 citations

TL;DR

This paper introduces ASlib, a standardized benchmark library for algorithm selection that provides a common format and repository for diverse algorithm selection scenarios across AI domains. By offering pre-collected performance data and a shared evaluation platform, ASlib reduces computational overhead and enables fair, reproducible comparison of algorithm selection methods, demonstrating significant performance gains through machine learning-based selection across 17 scenarios from six domains.

ABSTRACT

The task of algorithm selection involves choosing an algorithm from a set of algorithms on a per-instance basis in order to exploit the varying performance of algorithms over a set of instances. The algorithm selection problem is attracting increasing attention from researchers and practitioners in AI. Years of fruitful applications in a number of domains have resulted in a large amount of data, but the community lacks a standard format or repository for this data. This situation makes it difficult to share and compare different approaches effectively, as is done in other, more established fields. It also unnecessarily hinders new researchers who want to work in this area. To address this problem, we introduce a standardized format for representing algorithm selection scenarios and a repository that contains a growing number of data sets from the literature. Our format has been designed to be able to express a wide variety of different scenarios. Demonstrating the breadth and power of our platform, we describe a set of example experiments that build and evaluate algorithm selection models through a common interface. The results display the potential of algorithm selection to achieve significant performance improvements across a broad range of problems and algorithms.

Motivation & Objective

• To address the lack of standardized data formats and shared repositories for algorithm selection research, which hinders reproducibility and comparison.
• To lower the barrier to entry for new researchers by providing pre-collected performance data and a common interface for evaluating algorithm selectors.
• To enable cross-domain evaluation of algorithm selection techniques through a growing collection of standardized, reusable scenarios.
• To support ongoing, dynamic evaluation of algorithm selection systems without rigid submission deadlines or hardware constraints.
• To mitigate overfitting risks by enabling regular updates with new scenarios and standardized cross-validation splits.

Proposed method

• The paper proposes a standardized data format for representing algorithm selection scenarios, including solver performance on problem instances, features, and metadata.
• It introduces ASlib, a public online repository (aslib.net) hosting 17 algorithm selection scenarios from six domains, including SAT, CSP, and planning.
• Each scenario includes pre-computed performance data, cross-validation splits, and features for problem instances and algorithms.
• The platform supports automated exploratory data analysis and allows researchers to submit results and source code for verification.
• Performance is evaluated using multiple metrics (e.g., average solved instances, PAR10, MCP) to assess robustness and trade-offs.
• The system enables on-the-fly updates and continuous evaluation, reflecting the current state-of-the-art without fixed deadlines.

Experimental results

Research questions

• RQ1 Can a standardized benchmark library improve reproducibility and comparability in algorithm selection research?
• RQ2 How does the performance of machine learning-based algorithm selectors compare to standalone solvers across diverse, real-world scenarios?
• RQ3 To what extent can pre-collected performance data reduce the computational burden of evaluating algorithm selection methods?
• RQ4 How can ongoing, dynamic evaluation platforms help detect and prevent overfitting in algorithm selection systems?
• RQ5 What are the key challenges in benchmarking algorithm selection systems, and how can they be addressed through a shared infrastructure?

Key findings

• ASlib’s standardized format and repository enable researchers to evaluate algorithm selection methods without running solvers, significantly reducing computational cost.
• The platform supports dynamic, continuous evaluation with no fixed deadlines, allowing the latest systems and results to be integrated in real time.
• Performance evaluations using multiple metrics (e.g., PAR10, MCP) revealed trade-offs, such as algorithm schedules improving solved instances but increasing misclassification penalties.
• The study demonstrated that algorithm selection can achieve near-oracle performance, solving 92% of SAT instances when the best solver is chosen per instance, compared to 77% for the best standalone solver.
• The ICON Challenge on Algorithm Selection confirmed the effectiveness of state-of-the-art systems like zilla, which outperformed others across multiple metrics despite resource constraints.
• The inclusion of 17 diverse scenarios from six domains ensures that algorithm selection methods are tested under varied and realistic conditions, enhancing generalizability.