[Paper Review] Towards White-box Benchmarks for Algorithm Control

André Biedenkapp, H. Furkan Bozkurt+2 more|arXiv (Cornell University)|Jun 18, 2019

Machine Learning and Data Classification31 references4 citations

TL;DR

This paper introduces a novel framework for learning dynamic hyperparameter control in algorithms using reinforcement learning (RL) formulated as a contextual Markov decision process (MDP), where policies adapt to problem instances over time. It proposes three new white-box benchmarks to evaluate algorithm control and demonstrates that RL outperforms static black-box configuration, especially as policy complexity and instance heterogeneity increase.

ABSTRACT

The performance of many algorithms in the fields of hard combinatorial problem solving, machine learning or AI in general depends on tuned hyperparameter configurations. Automated methods have been proposed to alleviate users from the tedious and error-prone task of manually searching for performance-optimized configurations across a set of problem instances. However there is still a lot of untapped potential through adjusting an algorithm's hyperparameters online since different hyperparameters are potentially optimal at different stages of the algorithm. We formulate the problem of adjusting an algorithm's hyperparameters for a given instance on the fly as a contextual MDP, making reinforcement learning (RL) the prime candidate to solve the resulting algorithm control problem in a data-driven way. Furthermore, inspired by applications of algorithm configuration, we introduce new white-box benchmarks suitable to study algorithm control. We show that on short sequences, algorithm configuration is a valid choice, but that with increasing sequence length a black-box view on the problem quickly becomes infeasible and RL performs better.

Motivation & Objective

• Address the limitation of static hyperparameter configurations in iterative algorithms by enabling dynamic, instance-aware control.
• Formalize online hyperparameter adjustment as a contextual MDP to model instance-dependent policy learning.
• Develop new, transparent (white-box) benchmarks to evaluate algorithm control across diverse problem instances.
• Investigate the feasibility and performance of black-box optimization versus reinforcement learning for dynamic algorithm control.
• Demonstrate that RL generalizes better than static configuration on unseen or complex instance sets.

Proposed method

• Formulate algorithm control as a contextual MDP with states representing algorithm progress and instance features, actions as hyperparameter settings, and rewards based on performance metrics.
• Use deep Q-networks (DQN) with function approximation to learn policies in high-dimensional state-action spaces.
• Design three new white-box benchmarks: Sigmoid, SigmoidMVA, and a synthetic MVA problem, each with instance-specific optimal hyperparameter sequences.
• Train agents using deep RL (DQN), epsilon-greedy Q-learning, and black-box optimizers (SMAC, URS) on both training and test instance sets.
• Evaluate generalization by training on a finite set of instances and testing on unseen ones, measuring policy robustness and convergence speed.
• Apply smoothing and statistical averaging over 25 training runs to ensure reliable performance estimation with standard error reporting.

Experimental results

Research questions

• RQ1 Can reinforcement learning effectively learn dynamic hyperparameter policies that adapt to individual problem instances during algorithm execution?
• RQ2 How does the performance of black-box algorithm configuration compare to RL-based control as policy length and complexity increase?
• RQ3 To what extent can RL agents generalize from a finite set of training instances to unseen test instances in algorithm control tasks?
• RQ4 How do different exploration strategies (e.g., epsilon-greedy vs. random sampling) affect policy learning in dynamic algorithm control?
• RQ5 What role does state information play in enabling RL agents to outperform black-box optimizers in heterogeneous instance settings?

Key findings

• Reinforcement learning with function approximation (DQN) successfully learns dynamic hyperparameter policies on the Sigmoid and SigmoidMVA benchmarks, outperforming static black-box configuration as policy length increases.
• On short sequences, black-box optimization (e.g., SMAC) performs comparably to DQN, validating its feasibility for simple, fixed-length policies.
• Tabular Q-learning agents fail to generalize to unseen test instances due to overfitting and local optima, even when trained on a full training set of 100 instances.
• DQN generalizes effectively across unseen test instances, learning a robust policy that adapts to instance-specific optimal hyperparameter sequences.
• In high-dimensional action spaces (e.g., SigmoidMVA with 5 actions), black-box optimizers like SMAC struggle without state information, while DQN adapts effectively using state context.
• The proposed white-box benchmarks enable controlled evaluation of algorithm control, revealing the limitations of static configuration and the advantages of data-driven, dynamic policy learning.