[Paper Review] Winner Takes It All: Training Performant RL Populations for Combinatorial Optimization

Nathan Grinsztajn, Daniel Furelos-Blanco+3 more|arXiv (Cornell University)|Oct 7, 2022

Vehicle Routing Optimization Methods6 citations

TL;DR

This paper introduces Poppy, a reinforcement learning method that trains a population of complementary policies for combinatorial optimization by specializing agents on distinct problem instances through a performance-driven objective. By optimizing only the best-performing agent per instance during training, Poppy achieves state-of-the-art results on TSP, CVRP, 0-1 knapsack, and job-shop scheduling without explicit diversity supervision.

ABSTRACT

Applying reinforcement learning (RL) to combinatorial optimization problems is attractive as it removes the need for expert knowledge or pre-solved instances. However, it is unrealistic to expect an agent to solve these (often NP-)hard problems in a single shot at inference due to their inherent complexity. Thus, leading approaches often implement additional search strategies, from stochastic sampling and beam search to explicit fine-tuning. In this paper, we argue for the benefits of learning a population of complementary policies, which can be simultaneously rolled out at inference. To this end, we introduce Poppy, a simple training procedure for populations. Instead of relying on a predefined or hand-crafted notion of diversity, Poppy induces an unsupervised specialization targeted solely at maximizing the performance of the population. We show that Poppy produces a set of complementary policies, and obtains state-of-the-art RL results on four popular NP-hard problems: traveling salesman, capacitated vehicle routing, 0-1 knapsack, and job-shop scheduling.

Motivation & Objective

• To address the limitation of single-agent RL in solving NP-hard combinatorial optimization problems due to inherent solution space complexity.
• To overcome the bias of single policies in construction-based RL methods that restrict exploration to a single heuristic path.
• To develop a scalable, efficient training procedure that induces effective policy diversity without relying on handcrafted behavioral markers or external supervision.
• To demonstrate that performance-driven specialization leads to complementary policies that outperform existing RL-based solvers across multiple NP-hard problems.

Proposed method

• Poppy trains a population of agents by sharing a shared backbone network and specializing only lightweight policy heads.
• It uses a novel policy gradient objective that updates only the agent achieving the best performance on each problem instance during training.
• The method leverages a shared encoder with individualized policy heads, enabling efficient computation across the population.
• It applies REINFORCE with a state-value baseline and uses environment dynamics from JAX-based Jumanji for efficient simulation.
• The training objective implicitly promotes diversity by focusing on performance gaps across the population, without explicit behavioral regularization.
• The approach is applied to four problems: TSP, CVRP, 0-1 knapsack, and job-shop scheduling, using consistent architectures and hyperparameters across tasks.

Experimental results

Research questions

• RQ1 Can a population of RL agents trained via a performance-driven objective achieve better generalization and solution quality than single-agent methods in combinatorial optimization?
• RQ2 Does implicit specialization through performance-based training lead to effective, complementary policies without explicit diversity constraints?
• RQ3 How does Poppy compare to existing RL-based solvers in terms of solution quality and inference efficiency across diverse NP-hard problems?
• RQ4 Can the proposed method scale to large instances and maintain strong performance without additional search or fine-tuning?

Key findings

• Poppy achieves state-of-the-art performance on all four benchmark problems: TSP, CVRP, 0-1 knapsack, and job-shop scheduling.
• On TSP1000, Poppy with 16 trajectories achieves a solution gap of 71.7%, significantly outperforming POMO (120% gap) and EAS (114% gap).
• In time-performance trade-off analysis, Poppy consistently outperforms POMO across all settings, and increasing sampling to match Poppy’s performance is often intractable.
• For the 0-1 knapsack problem, training a population from scratch without pre-training a single decoder is effective and efficient, taking only minutes.
• The method achieves strong results on job-shop scheduling with 10x10 instances, minimizing makespan using a transformer-based actor-critic architecture with JAX-accelerated dynamics.
• Poppy’s performance is robust across problem sizes and does not require handcrafted diversity or search heuristics, relying solely on the population-level objective.