[Paper Review] Learning Collaborative Policies to Solve NP-hard Routing Problems

Min-Su Kim, Jinkyoo Park+1 more|arXiv (Cornell University)|Oct 26, 2021

Vehicle Routing Optimization MethodsEngineering33 references43 citations

TL;DR

Introduces a two-policy hierarchical DRL framework (seeder and reviser) to solve NP-hard routing problems by generating diverse seeds and iteratively revising them for quality, improving over single-policy DRL baselines across TSP, PCTSP, and CVRP.

ABSTRACT

Recently, deep reinforcement learning (DRL) frameworks have shown potential for solving NP-hard routing problems such as the traveling salesman problem (TSP) without problem-specific expert knowledge. Although DRL can be used to solve complex problems, DRL frameworks still struggle to compete with state-of-the-art heuristics showing a substantial performance gap. This paper proposes a novel hierarchical problem-solving strategy, termed learning collaborative policies (LCP), which can effectively find the near-optimum solution using two iterative DRL policies: the seeder and reviser. The seeder generates as diversified candidate solutions as possible (seeds) while being dedicated to exploring over the full combinatorial action space (i.e., sequence of assignment action). To this end, we train the seeder's policy using a simple yet effective entropy regularization reward to encourage the seeder to find diverse solutions. On the other hand, the reviser modifies each candidate solution generated by the seeder; it partitions the full trajectory into sub-tours and simultaneously revises each sub-tour to minimize its traveling distance. Thus, the reviser is trained to improve the candidate solution's quality, focusing on the reduced solution space (which is beneficial for exploitation). Extensive experiments demonstrate that the proposed two-policies collaboration scheme improves over single-policy DRL framework on various NP-hard routing problems, including TSP, prize collecting TSP (PCTSP), and capacitated vehicle routing problem (CVRP).

Motivation & Objective

• Motivate solving NP-hard routing problems without problem-specific heuristics using deep RL.
• Propose a hierarchical framework with two specialized policies to balance exploration and exploitation.
• Demonstrate that collaborative policies improve solution quality and speed on TSP, PCTSP, and CVRP.
• Show that entropy-driven seeding combined with parallel revising yields diverse yet refined solutions.

Proposed method

• Define a two-stage DRL framework: a seeder that generates diverse candidate tours by maximizing entropy over the full combinatorial action space; a reviser that partitions tours into sub-tours and optimizes each sub-tour in parallel to improve quality.
• Use entropy regularization to encourage seed diversity, approximating entropy via a time-weighted sum of segment-policy entropies.
• Train seeder and reviser separately, leveraging AM (Attention Model) as the underlying neural architecture for policy parameterization.
• In revision, convert full trajectories into multiple sub-tours, revise each segment with a shared context embedding adapted from the AM encoder, and aggregate updated full tours to select the best candidate.
• Provide a training Objective based on REINFORCE with rollout baseline, incorporating an entropy-based reward for seeding and a negative-length reward for revising.
• Report performance on TSP, PCTSP, and CVRP with varying node counts (N=20, 50, 100, 500), comparing against Gurobi, OR-Tools, Concorde, LKH3, and other DRL baselines.

Experimental results

Research questions

• RQ1 Can a two-policy DRL framework (seeder and reviser) outperform single-policy DRL methods on NP-hard routing problems?
• RQ2 Does entropy-regularized seeding improve solution diversity without sacrificing exploitable quality when combined with parallelized revising?
• RQ3 How does the LCP scheme perform across TSP, PCTSP, and CVRP in terms of solution quality and computation time under realistic budgets?

Key findings

• LCP outperforms single-policy DRL baselines (e.g., AM, DRL-2opt) on TSP, PCTSP, and CVRP for N up to 100 in measured experiments.
• Entropy-regularized seeding plus the reviser yields best performance, validated by ablation showing improvements over vanilla AM and other configurations.
• Reviser’s parallel sub-tour revision enables larger effective search spaces per inference and reduces total iterations required, speeding up convergence.
• LCP demonstrates favorable time-performance trade-offs, achieving a winner region in time budgets where it outperforms DRL solvers and some conventional heuristics.
• The approach remains architecture-agnostic and can be applied to other AM-style models or neural architectures.