[Paper Review] RL4CO: an Extensive Reinforcement Learning for Combinatorial Optimization Benchmark

Federico Berto, Chuanbo Hua+31 more|arXiv (Cornell University)|Jun 29, 2023

Software Engineering Research58 references4 citations

TL;DR

RL4CO introduces a unified, open-source benchmark for reinforcement learning in combinatorial optimization, featuring 27 problem environments and 23 state-of-the-art baselines. It enables modular, reproducible research by decoupling algorithm development from engineering overhead, with empirical studies showing that diverse training distributions significantly improve generalization across CO tasks.

ABSTRACT

Combinatorial optimization (CO) is fundamental to several real-world applications, from logistics and scheduling to hardware design and resource allocation. Deep reinforcement learning (RL) has recently shown significant benefits in solving CO problems, reducing reliance on domain expertise and improving computational efficiency. However, the absence of a unified benchmarking framework leads to inconsistent evaluations, limits reproducibility, and increases engineering overhead, raising barriers to adoption for new researchers. To address these challenges, we introduce RL4CO, a unified and extensive benchmark with in-depth library coverage of 27 CO problem environments and 23 state-of-the-art baselines. Built on efficient software libraries and best practices in implementation, RL4CO features modularized implementation and flexible configurations of diverse environments, policy architectures, RL algorithms, and utilities with extensive documentation. RL4CO helps researchers build on existing successes while exploring and developing their own designs, facilitating the entire research process by decoupling science from heavy engineering. We finally provide extensive benchmark studies to inspire new insights and future work. RL4CO has already attracted numerous researchers in the community and is open-sourced at https://github.com/ai4co/rl4co.

Motivation & Objective

• Address the lack of a unified benchmark for reinforcement learning in combinatorial optimization, which hinders reproducibility and consistent evaluation.
• Reduce engineering overhead and domain-specific tuning by providing a modular, well-documented software framework for RL-based CO research.
• Facilitate the development and comparison of new RL algorithms and neural architectures by standardizing environments, baselines, and evaluation protocols.
• Enable systematic study of generalization in neural combinatorial optimization through diverse training distributions and VRP variants.
• Support the community in building foundational models across CO domains by offering extensible, production-ready implementations.

Proposed method

• Design a modular, extensible software library built on best practices in RL and CO, supporting flexible configuration of environments, policy architectures, and RL algorithms.
• Implement 27 diverse combinatorial optimization environments, including TSP, VRP, CVRP, bin packing, and scheduling problems, with standardized interfaces.
• Integrate 23 state-of-the-art baselines, such as POMO, MTPOMO, MVMoE, and MVMoE-L, to enable fair and consistent comparisons.
• Support multiple VRP variants and coordinate distribution types (e.g., uniform, clustered, Gaussian) to study generalization across data distributions.
• Enable training and evaluation across mixed distributions and tasks via configurable data generation pipelines and curriculum learning setups.
• Provide comprehensive documentation, logging, and evaluation tools to streamline experimentation and ensure reproducibility.

Experimental results

Research questions

• RQ1 How does training on diverse data distributions (e.g., mixed sizes and coordinate types) affect generalization performance in combinatorial optimization?
• RQ2 To what extent do multi-task training across different VRP variants improve zero-shot generalization on unseen problem instances?
• RQ3 Can a unified benchmark framework reduce engineering overhead and accelerate research progress in neural combinatorial optimization?
• RQ4 How do different policy architectures (e.g., MVMoE, MTPOMO) compare in terms of solution quality and generalization across CO problems?
• RQ5 What is the impact of architectural and training diversity on generalization beyond the training distribution in CO tasks?

Key findings

• Training on mixed distributions (e.g., MDPOMO) significantly improves generalization on problem instances of similar size to the training set, reducing average gap by up to 3.8% on CVRPLib datasets.
• Multi-task training across diverse VRP variants (e.g., MTPOMO) achieves superior generalization on larger and more diverse distributions, outperforming single-task models by up to 13.9% gap reduction on Set X (100–1000 customers).
• The MVMoE-L model achieves the best performance on the X-n101-k25 benchmark with a 10.38% gap, demonstrating the effectiveness of mixture-of-experts in handling large-scale VRPs.
• Generalization performance improves markedly when training data includes multiple VRP variants and coordinate distributions, indicating shared foundational knowledge across tasks.
• The X-n819-k171 instance shows a 164.2% gap increase for the strongest baseline (MVMoE-L), highlighting the challenge of generalization to extreme scales and the need for more robust models.
• Models trained on mixed distributions (e.g., MDPOMO) achieve lower average gaps across all CVRPLib sets, with Set X showing a 13.9% gap using MVMoE-L, indicating strong zero-shot generalization.