[Paper Review] Learning 2-opt Heuristics for the Traveling Salesman Problem via Deep Reinforcement Learning

Paulo Roberto de Oliveira da Costa, Jason Rhuggenaath+2 more|arXiv (Cornell University)|Apr 3, 2020

Vehicle Routing Optimization Methods26 references71 citations

TL;DR

The paper trains a policy gradient-based deep RL model to learn 2-opt improvement moves for the Euclidean TSP, using a pointer-attention decoder and dual encoders to achieve near-optimal solutions from random starts. It demonstrates faster convergence and better results than prior DL methods and adapts to general k-opt moves.

ABSTRACT

Recent works using deep learning to solve the Traveling Salesman Problem (TSP) have focused on learning construction heuristics. Such approaches find TSP solutions of good quality but require additional procedures such as beam search and sampling to improve solutions and achieve state-of-the-art performance. However, few studies have focused on improvement heuristics, where a given solution is improved until reaching a near-optimal one. In this work, we propose to learn a local search heuristic based on 2-opt operators via deep reinforcement learning. We propose a policy gradient algorithm to learn a stochastic policy that selects 2-opt operations given a current solution. Moreover, we introduce a policy neural network that leverages a pointing attention mechanism, which unlike previous works, can be easily extended to more general k-opt moves. Our results show that the learned policies can improve even over random initial solutions and approach near-optimal solutions at a faster rate than previous state-of-the-art deep learning methods.

Motivation & Objective

• Motivate learning improvement heuristics for the TSP, focusing on 2-opt moves rather than construction heuristics.
• Develop a reinforcement learning framework that learns a stochastic policy for selecting 2-opt moves to improve solutions.
• Create a policy neural network with a pointing attention mechanism that can extend to general k-opt moves.
• Show that learned policies can start from poor solutions and near-optimally improve them faster than prior DL methods.

Proposed method

• Formulate 2-opt improvement for TSP as an MDP with state = (current tour, best-so-far tour).
• Use policy gradient to learn a stochastic policy over 2-opt moves with advantage-based updates.
• Employ a dual encoder architecture combining Graph Convolutional Network (GCN) and sequential RNN encodings to represent tours and edges.
• Decode actions via a pointing attention mechanism that outputs node indices for feasible 2-opt moves, enabling extension to higher k-opt moves.
• Train with an actor-critic-like objective including an entropy bonus to encourage exploration.

Experimental results

Research questions

• RQ1 Can a learned policy for 2-opt improvements reach near-optimal TSP tours more efficiently than prior DL methods?
• RQ2 How well do dual graph-and-sequence encoders plus a pointing decoder perform for 2-opt moves on Euclidean TSP?
• RQ3 Do learned 2-opt policies generalize from small to larger TSP instances and from random initial solutions?
• RQ4 What is the impact of longer planning horizons (more extended episode lengths) on policy performance?

Key findings

• Learned policies achieve near-optimal solutions starting from random initial tours across TSP sizes (20, 50, 100 nodes).
• Policies with longer episodes improve validation performance by considering longer planning horizons.
• On 512 test instances for TSP100, the method outperforms classical 2-opt heuristics with restarts and many prior DL approaches in solution quality and consistency.
• With 500, 1000, and 2000 sampling steps, the method attains gaps of 0.01%, 0.00%, and 0.00% (TSP20) and 0.36%, 0.21%, and 0.12% (TSP50), 1.84%, 1.26%, and 0.87% (TSP100) respectively in the reported comparisons.
• The approach is more sample-efficient than prior RL methods and can adapt to general k-opt moves while maintaining competitive runtimes.