[Paper Review] Reinforcement Learning Driven Heuristic Optimization

Qingpeng Cai, Will Hang+4 more|arXiv (Cornell University)|Jun 16, 2019

Scheduling and Optimization Algorithms17 references19 citations

TL;DR

This paper proposes RLHO, a framework that uses reinforcement learning (PPO) to generate high-quality initial solutions for heuristic optimization algorithms like simulated annealing (SA), significantly improving performance on combinatorial problems. By training the RL agent using the final solution quality from SA as a reward signal, RLHO enables the agent to learn initialization strategies that make SA more effective, outperforming both pure RL and random initialization across multiple bin packing problem sizes.

ABSTRACT

Heuristic algorithms such as simulated annealing, Concorde, and METIS are effective and widely used approaches to find solutions to combinatorial optimization problems. However, they are limited by the high sample complexity required to reach a reasonable solution from a cold-start. In this paper, we introduce a novel framework to generate better initial solutions for heuristic algorithms using reinforcement learning (RL), named RLHO. We augment the ability of heuristic algorithms to greedily improve upon an existing initial solution generated by RL, and demonstrate novel results where RL is able to leverage the performance of heuristics as a learning signal to generate better initialization. We apply this framework to Proximal Policy Optimization (PPO) and Simulated Annealing (SA). We conduct a series of experiments on the well-known NP-complete bin packing problem, and show that the RLHO method outperforms our baselines. We show that on the bin packing problem, RL can learn to help heuristics perform even better, allowing us to combine the best parts of both approaches.

Motivation & Objective

• To address the high sample complexity of heuristic algorithms like simulated annealing that start from random initial solutions.
• To improve the performance of heuristic optimization by leveraging reinforcement learning to generate superior initial solutions.
• To enable RL to learn from the performance of heuristic algorithms by using the final solution quality as a reward signal.
• To demonstrate that combining RL with heuristics yields better results than using either method alone on NP-complete problems like bin packing.

Proposed method

• The RLHO framework alternates between generating an initial solution via a PPO agent and refining it using simulated annealing (SA).
• The RL agent is trained using the final solution quality from SA as a dense, informative reward signal, which guides policy improvement.
• The RL agent observes the problem state and takes actions to assign items to bins, producing an initial solution for SA.
• The SA component performs greedy, stochastic local search from the RL-generated initial solution, improving it through perturbations and acceptance based on a temperature schedule.
• The training loop alternates between RL rollouts and SA optimization, allowing the RL agent to explore better states via the heuristic's hill-climbing behavior.
• The framework is evaluated on the bin packing problem with varying instance sizes, using PPO for RL and SA for heuristic optimization.

Experimental results

Research questions

• RQ1 Can reinforcement learning learn to generate better initial solutions for heuristic optimization algorithms like simulated annealing?
• RQ2 Does using the final solution quality from a heuristic as a reward signal improve the performance of the RL agent in combinatorial optimization?
• RQ3 Can the combination of RL and heuristics outperform both pure RL and pure heuristic methods on NP-complete problems?
• RQ4 Does the RL agent learn initialization strategies that remain effective even when the heuristic runs for significantly longer durations than during training?

Key findings

• For the 1000-item bin packing problem, RLHO reduced the average number of bins used from 734 (random initialization) to 711 when SA ran for 50,000 steps.
• With 5000 SA steps, RLHO achieved an average of 714 bins for n=1000, compared to 734 with random initialization, showing a consistent 2.7% improvement.
• When SA was allowed to run until convergence (millions of steps), the initialization generated by RLHO trained with only 5000 SA steps still outperformed random initialization.
• In alternating optimization, RLHO achieved a 2.1% improvement over pure PPO on the 1000-item problem after 2000 training iterations, demonstrating better exploration.
• The RL agent learned to generate initial solutions that were more amenable to SA’s hill-climbing process, even though RL alone did not converge to optimal solutions.
• The performance gap between RLHO and random initialization increased with problem size, indicating that RLHO scales better on larger, more complex instances.