[Paper Review] Reinforcement Learning Enhanced Quantum-inspired Algorithm for Combinatorial Optimization

Dmitrii Beloborodov, Alexander E. Ulanov+3 more|arXiv (Cornell University)|Feb 11, 2020

Quantum Computing Algorithms and Architecture31 references4 citations

TL;DR

This paper proposes a reinforcement learning (RL) agent that dynamically tunes the regularization parameter in the SimCIM quantum-inspired algorithm for Ising model optimization, using a novel Rescaled Ranked Reward (R3) method to escape local optima. The approach significantly outperforms baseline heuristics and black-box optimization (e.g., CMA-ES), achieving high-probability discovery of high-quality solutions, especially when combined with transfer learning from pre-trained agents on random problems.

ABSTRACT

Quantum hardware and quantum-inspired algorithms are becoming increasingly popular for combinatorial optimization. However, these algorithms may require careful hyperparameter tuning for each problem instance. We use a reinforcement learning agent in conjunction with a quantum-inspired algorithm to solve the Ising energy minimization problem, which is equivalent to the Maximum Cut problem. The agent controls the algorithm by tuning one of its parameters with the goal of improving recently seen solutions. We propose a new Rescaled Ranked Reward (R3) method that enables stable single-player version of self-play training that helps the agent to escape local optima. The training on any problem instance can be accelerated by applying transfer learning from an agent trained on randomly generated problems. Our approach allows sampling high-quality solutions to the Ising problem with high probability and outperforms both baseline heuristics and a black-box hyperparameter optimization approach.

Motivation & Objective

• To address the challenge of hyperparameter tuning in quantum-inspired combinatorial optimization algorithms like SimCIM, which requires problem-specific configuration for optimal performance.
• To improve solution quality and convergence speed in solving the Ising energy minimization problem, equivalent to the Max-Cut problem, by automating parameter adaptation using reinforcement learning.
• To overcome the difficulty of local optima in Ising optimization, where many suboptimal solutions have similar energy values, by designing a reward mechanism that encourages exploration beyond these traps.
• To accelerate training on new problem instances by enabling transfer learning from a pre-trained agent on randomly generated problems, improving sample efficiency.
• To generalize the approach beyond SimCIM and the Ising model to other continuous relaxation-based discrete optimization algorithms.

Proposed method

• A reinforcement learning agent is trained to control the regularization (gain-loss) function in the SimCIM algorithm, which governs the dynamic evolution of the solution during optimization.
• The Rescaled Ranked Reward (R3) mechanism assigns rewards based on the relative ranking of the current solution's cut value among recent trials, with lower rewards for frequently visited local optima and fixed rewards for better solutions.
• R3 enables stable single-player self-play training by making rewards sensitive to solution frequency, thus encouraging the agent to explore new, higher-quality regions of the solution space.
• Transfer learning is applied by pre-training the RL agent on a diverse set of randomly generated Ising problem instances, followed by fine-tuning on a target problem using feature-wise linear modulation (FiLM) to adapt to the new problem's parameters.
• The agent's policy network is trained using deep reinforcement learning with a critic network to estimate value functions, and the training is accelerated via GPU implementation.
• The method is evaluated on standard Gset benchmark instances, comparing solution quality and convergence speed against fixed-parameter SimCIM and CMA-ES.

Experimental results

Research questions

• RQ1 Can a reinforcement learning agent effectively automate the tuning of the regularization parameter in the SimCIM algorithm to improve solution quality on Ising optimization problems?
• RQ2 Does the proposed Rescaled Ranked Reward (R3) mechanism enable stable training and effective escape from local optima compared to standard reward schemes like R2?
• RQ3 To what extent can transfer learning from a pre-trained agent on random problems accelerate fine-tuning and improve performance on unseen, real-world problem instances?
• RQ4 How does the RL-enhanced SimCIM approach compare in performance and sample efficiency to traditional heuristics and black-box hyperparameter optimization methods like CMA-ES?
• RQ5 Can the proposed method generalize beyond SimCIM and the Ising model to other continuous relaxation-based discrete optimization algorithms?

Key findings

• The R3 method enables stable training and effective escape from local optima, as evidenced by a significant drop in value loss after the agent discovers better solutions, unlike the R2 baseline which suffers from unstable training due to random rewards for local optima.
• The pre-trained agent with both FiLM and R3 outperforms ablated versions, showing faster convergence and higher solution quality, with the transfer learning approach reducing the number of episodes needed to solve problems.
• On the G2 instance from the Gset benchmark, the R3-based agent achieved a 0.80 fraction of solved instances after 500 fine-tuning episodes, outperforming the baseline SimCIM with manually tuned parameters (0.70) and CMA-ES (0.53).
• The ablation study confirms that both FiLM and R3 are essential: the agent without either component shows significantly slower convergence and lower solution quality.
• The RL-enhanced SimCIM achieves a higher probability of sampling high-quality solutions than both heuristic baselines and CMA-ES, demonstrating robustness and adaptability across diverse problem instances.
• The approach is generalizable and not limited to SimCIM or the Ising model, as it can be applied to any algorithm based on continuous relaxation of discrete optimization problems.