[Paper Review] Automated Dynamic Algorithm Configuration

Steven Adriænsen, André Biedenkapp+5 more|arXiv (Cornell University)|May 27, 2022

Metaheuristic Optimization Algorithms Research4 citations

TL;DR

This paper introduces automated dynamic algorithm configuration (DAC), a novel framework that uses data-driven, reinforcement learning-based methods to automatically learn dynamic parameter adaptation policies during algorithm execution. It demonstrates significant performance gains over static configurations in evolutionary optimization, AI planning, and machine learning, establishing DAC as a foundational approach for adaptive, high-performing algorithms.

ABSTRACT

The performance of an algorithm often critically depends on its parameter configuration. While a variety of automated algorithm configuration methods have been proposed to relieve users from the tedious and error-prone task of manually tuning parameters, there is still a lot of untapped potential as the learned configuration is static, i.e., parameter settings remain fixed throughout the run. However, it has been shown that some algorithm parameters are best adjusted dynamically during execution, e.g., to adapt to the current part of the optimization landscape. Thus far, this is most commonly achieved through hand-crafted heuristics. A promising recent alternative is to automatically learn such dynamic parameter adaptation policies from data. In this article, we give the first comprehensive account of this new field of automated dynamic algorithm configuration (DAC), present a series of recent advances, and provide a solid foundation for future research in this field. Specifically, we (i) situate DAC in the broader historical context of AI research; (ii) formalize DAC as a computational problem; (iii) identify the methods used in prior-art to tackle this problem; (iv) conduct empirical case studies for using DAC in evolutionary optimization, AI planning, and machine learning.

Motivation & Objective

• To formalize automated dynamic algorithm configuration (DAC) as a new meta-algorithmic framework that learns dynamic parameter policies from data.
• To address the limitations of static algorithm configuration, where fixed parameter settings fail to adapt to changing problem landscapes during execution.
• To overcome the reliance on manual, heuristic-based dynamic parameter tuning by enabling automated, data-driven policy learning.
• To provide a comprehensive foundation for DAC, including formal problem definition, methodological review, and empirical validation across diverse domains.
• To identify key research challenges and future directions, such as online learning, warm-starting, and expert-in-the-loop DAC

Proposed method

• Formalize DAC as a contextual reinforcement learning problem, where policies adapt parameters based on real-time state observations during algorithm execution.
• Apply deep reinforcement learning agents to learn dynamic configuration policies that maximize performance across a distribution of problem instances.
• Use a unified benchmark framework (DACBench) to evaluate DAC methods across multiple domains, including evolutionary computation, AI planning, and machine learning.
• Integrate domain-specific knowledge into the configuration space by defining parameter bounds, performance metrics, and instance distributions relevant to each application.
• Enable policy transfer and warm-starting by leveraging existing heuristic or learned policies as initialization for automated DAC.
• Design a modular, extensible software stack to support comparison and extension of DAC methods, including standardized evaluation protocols

Experimental results

Research questions

• RQ1 How can dynamic algorithm configuration be formally defined as a computational problem amenable to automated, data-driven learning?
• RQ2 To what extent can automated DAC outperform both static configuration and manually designed dynamic heuristics in real-world AI applications?
• RQ3 What are the key methodological challenges in learning effective dynamic configuration policies, and how can they be addressed with modern machine learning techniques?
• RQ4 How can domain experts be integrated into the DAC process to guide configuration space design, performance metrics, and policy priors?
• RQ5 What role can online, continual learning play in refining DAC policies during deployment, based on real-time feedback from test instances?

Key findings

• Automated DAC using contextual reinforcement learning can outperform both static configuration and manually designed dynamic heuristics in evolutionary optimization, particularly in step-size adaptation within CMA-ES.
• Empirical case studies in evolutionary optimization, AI planning, and machine learning demonstrate that dynamic policies learned via DAC consistently improve performance over static or hand-crafted alternatives.
• The proposed DAC framework enables automatic discovery of complex, context-dependent parameter adaptation strategies that are difficult to engineer manually.
• Warm-starting DAC with existing heuristic policies significantly accelerates convergence and improves final policy quality, suggesting strong potential for practical deployment.
• The lack of standardized benchmarks and evaluation protocols has hindered method comparison; the paper advocates for DACBench and a shared library of DAC methods to enable reproducible research.
• Future directions such as online DAC, expert-in-the-loop configuration, and partial reconfiguration show strong potential to further enhance adaptability and usability of DAC