[Paper Review] A new benchmark set for Traveling salesman problem and Hamiltonian cycle problem

Pouya Baniasadi, Vladimir Ejov+2 more|arXiv (Cornell University)|Jun 25, 2018

Vehicle Routing Optimization Methods23 references6 citations

TL;DR

This paper introduces a new benchmark set for the Traveling Salesman Problem (TSP) and Hamiltonian Cycle Problem (HCP) based on small, structurally complex instances derived from difficult HCP examples. By focusing on outlier instances that challenge state-of-the-art TSP solvers—particularly those with high symmetry and low Hamiltonian cycle counts—it reveals algorithmic weaknesses and demonstrates that Concorde, LKH, and SLH each struggle in distinct ways, with LKH excelling in speed but failing on sparse cycle structures.

ABSTRACT

We present a benchmark set for Traveling salesman problem (TSP) with characteristics that are different from the existing benchmark sets. In particular, we focus on small instances which prove to be challenging for one or more state-of-the-art TSP algorithms. These instances are based on difficult instances of Hamiltonian cycle problem (HCP). This includes instances from literature, specially modified randomly generated instances, and instances arising from the conversion of other difficult problems to HCP. We demonstrate that such benchmark instances are helpful in understanding the weaknesses and strengths of algorithms. In particular, we conduct a benchmarking exercise for this new benchmark set totalling over five years of CPU time, comparing the TSP algorithms Concorde, Chained Lin-Kernighan, and LKH. We also include the HCP heuristic SLH in the benchmarking exercise. A discussion about the benefits of specifically considering outlying instances, and in particular instances which are unusually difficult relative to size, is also included.

Motivation & Objective

• To address the lack of meaningful benchmark sets for TSP and HCP that stress algorithmic weaknesses beyond generic performance.
• To identify and collect small, structurally difficult instances that are outliers in terms of algorithmic difficulty, especially for state-of-the-art solvers.
• To evaluate the performance of leading TSP and HCP algorithms—Concorde, Chained Lin-Kernighan, LKH, and SLH—on these hard instances to expose comparative strengths and weaknesses.
• To demonstrate that difficult HCP instances, when converted to TSP, produce TSP instances that stress existing TSP solvers in novel ways not captured by standard benchmarks like TSPLIB.

Proposed method

• The benchmark set is constructed from three sources: existing difficult HCP instances from the literature, specially modified randomly generated HCP instances, and HCP conversions of other hard combinatorial problems.
• Instances are converted into TSP format by assigning edge weights based on the HCP structure, preserving the underlying difficulty.
• A comprehensive benchmarking campaign was conducted using over five years of CPU time across 24 different instance types, evaluating four algorithms: Concorde (exact), Chained Lin-Kernighan, LKH (heuristic), and SLH (HCP-specific heuristic).
• Performance was measured by solution quality, runtime, and success rate (optimal solution found within 24 hours or 4GB memory limit), with failure types clearly marked.
• The study focuses on identifying structural features—such as high symmetry and low number of Hamiltonian cycles—that correlate with algorithmic difficulty.
• Statistical and qualitative analysis of failure patterns across algorithms was used to infer algorithmic limitations and structural difficulty drivers.

Experimental results

Research questions

• RQ1 What structural features in small TSP and HCP instances cause state-of-the-art solvers to fail or perform poorly?
• RQ2 How do the performance characteristics of Concorde, LKH, Chained Lin-Kernighan, and SLH differ on hard outlier instances not found in standard benchmarks?
• RQ3 To what extent do high symmetry and low number of Hamiltonian cycles contribute to algorithmic difficulty in TSP and HCP?
• RQ4 Can HCP instances be effectively repurposed as challenging TSP benchmarks, and if so, which algorithmic weaknesses do they expose?
• RQ5 What insights can be gained about the nature of TSP difficulty by studying small, hard instances derived from HCP?

Key findings

• Concorde, as an exact algorithm, was the most robust overall but occasionally failed to terminate within 24 hours on highly symmetric instances, possibly indicating implementation bugs.
• LKH outperformed Chained Lin-Kernighan in all tested instances, being significantly faster and more effective, but struggled when the number of Hamiltonian cycles was low.
• Chained Lin-Kernighan showed poor performance on instances with low cycle counts, indicating that Lin-Kernighan-based heuristics are sensitive to structural sparsity in the solution space.
• SLH was highly reliable for finding Hamiltonian cycles, even on complex graphs, but required excessive memory for large instances and was slower than Concorde and LKH.
• The most challenging instances were those combining high symmetry with a low number of Hamiltonian cycles, which consistently stymied multiple algorithms, especially LKH and CLK.
• Instances like II_3972 and II_7932 required over 56,000 seconds and exceeded 4GB memory, respectively, indicating extreme difficulty even for modern solvers.