[Paper Review] POMO: Policy Optimization with Multiple Optima for Reinforcement Learning

Yeong‐Dae Kwon, Jinho Choo+4 more|arXiv (Cornell University)|Oct 30, 2020

Reinforcement Learning in Robotics29 references135 citations

TL;DR

POMO trains a reinforcement learning solver for combinatorial optimization by exploring multiple starting points and using a shared baseline, achieving state-of-the-art results on TSP, CVRP, and KP with faster inference.

ABSTRACT

In neural combinatorial optimization (CO), reinforcement learning (RL) can turn a deep neural net into a fast, powerful heuristic solver of NP-hard problems. This approach has a great potential in practical applications because it allows near-optimal solutions to be found without expert guides armed with substantial domain knowledge. We introduce Policy Optimization with Multiple Optima (POMO), an end-to-end approach for building such a heuristic solver. POMO is applicable to a wide range of CO problems. It is designed to exploit the symmetries in the representation of a CO solution. POMO uses a modified REINFORCE algorithm that forces diverse rollouts towards all optimal solutions. Empirically, the low-variance baseline of POMO makes RL training fast and stable, and it is more resistant to local minima compared to previous approaches. We also introduce a new augmentation-based inference method, which accompanies POMO nicely. We demonstrate the effectiveness of POMO by solving three popular NP-hard problems, namely, traveling salesman (TSP), capacitated vehicle routing (CVRP), and 0-1 knapsack (KP). For all three, our solver based on POMO shows a significant improvement in performance over all recent learned heuristics. In particular, we achieve the optimality gap of 0.14% with TSP100 while reducing inference time by more than an order of magnitude.

Motivation & Objective

• Identify symmetries and multiple optima in reinforcement learning for combinatorial optimization (CO).
• Develop a training method that exploits multiple optimal representations to improve exploration and stability.
• Propose an efficient inference approach leveraging multiple greedy rollouts and instance augmentation.
• Demonstrate POMO on three NP-hard problems (TSP, CVRP, KP) with improved gap and speed over prior neural RL methods.

Proposed method

• Introduce POMO by designating N different starting nodes to generate multiple solution rollouts in parallel for each instance.
• Use a shared baseline b_shared(s) computed as the average return across N trajectories to reduce gradient variance and resist premature convergence.
• Apply REINFORCE with the multi-start trajectory scheme and the shared baseline to optimize policy parameters.
• Adopt a multiple-greedy-trajectory inference method where N greedy trajectories from different starts are evaluated to select the best solution.
• Incorporate instance augmentation during inference by applying problem-transformations (e.g., coordinate rotations/transforms) to generate additional greedy rollouts for the same instance.
• Demonstrate compatibility with the Attention Model policy network and implement parallel rollout generation within the network’s decoder.

Experimental results

Research questions

• RQ1 Can exploiting symmetry and multiple optima improve exploration and stability in RL for CO problems?
• RQ2 Does a shared baseline across multiple trajectories reduce gradient variance and mitigate local minima better than greedy-rollout baselines?
• RQ3 How do multi-start greedy inferences and instance augmentation affect solution quality and inference time for TSP, CVRP, and KP?

Key findings

• POMO achieves near-optimal solutions with very small optimality gaps on TSP100 (0.14%) using multiple greedy rollouts and augmentation, outperforming prior learned heuristics in both quality and speed.
• Training with multiple starting nodes and a shared baseline results in more stable, sample-efficient policy learning and reduced susceptibility to local minima compared to greddy-rollout baselines.
• For inference, multi-start greedy rollouts (with instance augmentation) frequently yield better solutions than single-trajectory or purely sampled approaches, and can dramatically reduce inference time.
• POMO attains competitive or superior performance across three problems (TSP, CVRP, KP) using the same neural network and training procedure, highlighting its problem-agnostic applicability.
• The approach shows significant speedups in inference (e.g., TSP100) while maintaining or improving solution quality relative to contemporary neural RL methods.