[Paper Review] Bandit-Based Random Mutation Hill-Climbing

Jialin Liu, Diego Pérez-Liébana+1 more|arXiv (Cornell University)|Jun 20, 2016

Advanced Bandit Algorithms Research21 references3 citations

TL;DR

This paper proposes a bandit-based Random Mutation Hill-Climbing (RMHC) algorithm that replaces random neighbor selection with multi-armed bandit selection units to balance exploration and exploitation. It significantly outperforms standard RMHC, reducing fitness evaluations by up to a factor of ten in noisy OneMax and Royal Road problems, especially when using a resampling number of 2, with linear scaling relative to problem size.

ABSTRACT

The Random Mutation Hill-Climbing algorithm is a direct search technique mostly used in discrete domains. It repeats the process of randomly selecting a neighbour of a best-so-far solution and accepts the neighbour if it is better than or equal to it. In this work, we propose to use a novel method to select the neighbour solution using a set of independent multi- armed bandit-style selection units which results in a bandit-based Random Mutation Hill-Climbing algorithm. The new algorithm significantly outperforms Random Mutation Hill-Climbing in both OneMax (in noise-free and noisy cases) and Royal Road problems (in the noise-free case). The algorithm shows particular promise for discrete optimisation problems where each fitness evaluation is expensive.

Motivation & Objective

• To address the inefficiency of standard RMHC in noisy and complex discrete optimization problems where fitness evaluations are expensive.
• To improve exploration-exploitation trade-off in hill-climbing by integrating multi-armed bandit selection mechanisms.
• To develop a scalable and robust optimization method suitable for real-world applications with noisy or uncertain fitness evaluations.
• To evaluate the performance of the bandit-based RMHC on standard benchmark problems, including OneMax and Royal Road functions.
• To demonstrate that bandit-based selection leads to faster convergence and reduced evaluation overhead compared to random neighbor selection.

Proposed method

• The algorithm models each gene in a binary string as a 2-armed bandit, where arms correspond to flipping the bit to 0 or 1.
• It uses the Upper Confidence Bound (UCB) algorithm to select the next neighbor, balancing exploration and exploitation via an urgency term.
• Fitness evaluations are stored and updated per gene, with resampling used to mitigate noise in evaluations.
• The best-so-far solution is maintained, and the bandit units are updated based on the fitness gain from each mutation.
• The method dynamically prioritizes promising gene values while exploring under-sampled options through the UCB exploration term.
• The algorithm is applied to OneMax and Royal Road problems with varying problem dimensions and block sizes.

Experimental results

Research questions

• RQ1 Can a bandit-based selection mechanism improve the performance of Random Mutation Hill-Climbing in noisy and complex discrete optimization problems?
• RQ2 How does the bandit-based RMHC compare to standard RMHC in terms of fitness evaluation efficiency on OneMax and Royal Road functions?
• RQ3 What is the optimal resampling number for bandit-based RMHC in noisy environments, and how does it affect convergence?
• RQ4 Does the bandit-based approach scale linearly with problem dimension, and how does it handle delayed fitness rewards in credit assignment?
• RQ5 Can the bandit mechanism reduce the number of fitness evaluations required to reach the optimum, especially in noisy settings?

Key findings

• The bandit-based RMHC reduced fitness evaluations by up to a factor of ten compared to standard RMHC in noisy OneMax problems.
• With a resampling number of 2, the total number of fitness evaluations scaled almost linearly with problem dimension in both OneMax and Royal Road problems.
• For an 8x8 Royal Road problem, the bandit-based RMHC used only half the number of function evaluations compared to the most efficient RMHC variant reported by Mitchell et al.
• The algorithm maintained stable performance across different problem sizes and block configurations, outperforming standard RMHC in both noise-free and noisy settings.
• The UCB-based selection mechanism effectively balanced exploration and exploitation, preventing premature convergence even in noisy environments.
• The ratio of average fitness evaluations to problem dimension remained stable under bandit-based selection, unlike standard RMHC which showed increasing ratios with smaller resampling numbers.