[Paper Review] Learning Improvement Heuristics for Solving Routing Problems

Yaoxin Wu, Wen Song+3 more|arXiv (Cornell University)|Dec 12, 2019

Vehicle Routing Optimization Methods48 references49 citations

TL;DR

The paper trains a self-attention based reinforcement learning policy to learn improvement heuristics for routing problems, outperforming prior DL-based methods on TSP and CVRP and generalizing across sizes and datasets.

ABSTRACT

Recent studies in using deep learning to solve routing problems focus on construction heuristics, the solutions of which are still far from optimality. Improvement heuristics have great potential to narrow this gap by iteratively refining a solution. However, classic improvement heuristics are all guided by hand-crafted rules which may limit their performance. In this paper, we propose a deep reinforcement learning framework to learn the improvement heuristics for routing problems. We design a self-attention based deep architecture as the policy network to guide the selection of next solution. We apply our method to two important routing problems, i.e. travelling salesman problem (TSP) and capacitated vehicle routing problem (CVRP). Experiments show that our method outperforms state-of-the-art deep learning based approaches. The learned policies are more effective than the traditional hand-crafted ones, and can be further enhanced by simple diversifying strategies. Moreover, the policies generalize well to different problem sizes, initial solutions and even real-world dataset.

Motivation & Objective

• Motivate improving routing problem solutions beyond construction heuristics by learning improvement operators.
• Propose an RL framework that directly learns the policy for selecting next improvements in neighborhood search.
• Develop a self-attention based policy network to handle pairwise local operators like 2-opt and node swap.
• Apply the framework to TSP and CVRP and compare against state-of-the-art baselines.
• Demonstrate generalization to different problem sizes, initial solutions, and real-world datasets.

Proposed method

• Formulate improvement heuristics as a continuing MDP with states as current solutions and actions as node pairs to apply pairwise operators.
• Reward is the incremental improvement of the best incumbent solution found so far, encouraging continual betterment.
• Use a self-attention based policy network to compute a probability matrix for selecting node pairs to operate on, with a node embedding and node-pair selection module.
• In node embedding, incorporate positional encodings and self-attention to capture sequence information.
• In node pair selection, fuse global graph information with node embeddings and apply a masked softmax over feasible pairs.
• Train with an actor-critic algorithm (REINFORCE with bootstrapped critic) using n-step returns and a continuing task setup.

Experimental results

Research questions

• RQ1 Can a learned policy for improvement heuristics outperform hand-crafted rules in neighborhood search for routing problems?
• RQ2 How well does a self-attention based policy generalize across different problem sizes and initial solutions for TSP and CVRP?
• RQ3 Do learned improvement policies transfer to real-world datasets beyond synthetic instances?
• RQ4 What impact do different pairwise operators (e.g., 2-opt, node swap) have on the performance of the learned policy?
• RQ5 Can simple diversification strategies further enhance the quality of solutions produced by the learned policy?

Key findings

• The learned improvement policy significantly outperforms state-of-the-art DL-based approaches on TSP and CVRP.
• Policies learned from the RL framework beat traditional hand-crafted rules (first-improvement and best-improvement) across instance sizes.
• Increasing the allowed step limit T yields smaller optimality gaps, approaching Concorde-like quality on small TSP instances and outperforming several baselines on larger CVRP instances.
• The method achieves competitive runtimes with other DL-based methods and shows better scalability with problem size than some baselines.
• Diversifying via multi-run or multi-policy strategies further improves solution quality, with multi-run generally offering stronger gains than multi-policy for a fixed run count.
• The policies generalize to different problem sizes, initial solutions, and real-world datasets.